SherbyNet - /page/wikibook/entry.mu

<!DOCTYPE html>

<title>Regularization (mathematics)</title>

</head>

Regularization (mathematics)

</h1>

</header>

In <a href="Mathematics" title="Mathematics">mathematics</a>, <a href="Statistics" title="Statistics">statistics</a>, <a href="Mathematical_finance" title="Mathematical finance">finance</a>,<a href="#cite_note-1">[1]</a> and <a href="Computer_science" title="Computer science">computer science</a>, particularly in <a href="Machine_learning" title="Machine learning">machine learning</a> and <a href="Inverse_problem" title="Inverse problem">inverse problems</a>, regularization is a process that converts the <a href="Problem_solving" title="Problem solving">answer to a problem</a> to a simpler one. It is often used in solving <a href="Ill-posed_problems" class="mw-redirect" title="Ill-posed problems">ill-posed problems</a> or to prevent <a href="Overfitting" title="Overfitting">overfitting</a>.<a href="#cite_note-2">[2]</a>

Although regularization procedures can be divided in many ways, the following delineation is particularly helpful:

<ul><li>Explicit regularization is regularization whenever one explicitly adds a term to the optimization problem. These terms could be <a href="Prior_probability" title="Prior probability">priors</a>, penalties, or constraints. Explicit regularization is commonly employed with ill-posed optimization problems. The regularization term, or penalty, imposes a cost on the optimization function to make the optimal solution unique.</li>

<li>Implicit regularization is all other forms of regularization. This includes, for example, early stopping, using a robust loss function, and discarding outliers. Implicit regularization is essentially ubiquitous in modern machine learning approaches, including <a href="Stochastic_gradient_descent" title="Stochastic gradient descent">stochastic gradient descent</a> for training <a href="Deep_neural_networks" class="mw-redirect" title="Deep neural networks">deep neural networks</a>, and <a href="Ensemble_methods" class="mw-redirect" title="Ensemble methods">ensemble methods</a> (such as <a href="Random_forest" title="Random forest">random forests</a> and <a href="Gradient_boosted_trees" class="mw-redirect" title="Gradient boosted trees">gradient boosted trees</a>).</li></ul>

In explicit regularization, independent of the problem or model, there is always a data term, that corresponds to a likelihood of the measurement, and a regularization term that corresponds to a prior. By combining both using <a href="Bayesian_statistics" title="Bayesian statistics">Bayesian statistics</a>, one can compute a posterior, that includes both information sources and therefore stabilizes the estimation process. By trading off both objectives, one chooses to be more aligned to the data or to enforce regularization (to prevent overfitting). There is a whole research branch dealing with all possible regularizations. In practice, one usually tries a specific regularization and then figures out the probability density that corresponds to that regularization to justify the choice. It can also be physically motivated by common sense or intuition.

In <a href="Supervised_learning" title="Supervised learning">machine learning</a>, the data term corresponds to the training data and the regularization is either the choice of the model or modifications to the algorithm. It is always intended to reduce the <a href="Generalization_error" title="Generalization error">generalization error</a>, i.e. the error score with the trained model on the evaluation set (testing data) and not the training data.<a href="#cite_note-3">[3]</a>

One of the earliest uses of regularization is <a href="Tikhonov_regularization" class="mw-redirect" title="Tikhonov regularization">Tikhonov regularization</a> (ridge regression), related to the method of least squares.

<div class="mw-heading mw-heading2"><h2 id="Regularization_in_machine_learning">Regularization in machine learning</h2></div>

In <a href="Machine_learning" title="Machine learning">machine learning</a>, a key challenge is enabling models to accurately predict outcomes on unseen data, not just on familiar training data. Regularization is crucial for addressing <a href="Overfitting" title="Overfitting">overfitting</a>—where a model memorizes training data details but cannot generalize to new data. The goal of regularization is to encourage models to learn the broader patterns within the data rather than memorizing it. Techniques like <a href="Early_stopping" title="Early stopping">early stopping</a>, L1 and <a href="L2_regularization" class="mw-redirect" title="L2 regularization">L2 regularization</a>, and <a href="Dropout_(neural_networks)" class="mw-redirect" title="Dropout (neural networks)">dropout</a> are designed to prevent overfitting and underfitting, thereby enhancing the model's ability to adapt to and perform well with new data, thus improving model generalization.<a href="#cite_note-:1-4">[4]</a>

<div class="mw-heading mw-heading3"><h3 id="Early_Stopping">Early Stopping</h3></div>

Stops training when validation performance deteriorates, preventing overfitting by halting before the model memorizes training data.<a href="#cite_note-:1-4">[4]</a>

<div class="mw-heading mw-heading3"><h3 id="L1_and_L2_Regularization">L1 and L2 Regularization</h3></div>

Adds penalty terms to the cost function to discourage complex models:

<ul><li>L1 regularization (also called <a href="Lasso_(statistics)" title="Lasso (statistics)">LASSO</a>) leads to sparse models by adding a penalty based on the absolute value of coefficients.</li>

<li>L2 regularization (also called <a href="Ridge_regression" title="Ridge regression">ridge regression</a>) encourages smaller, more evenly distributed weights by adding a penalty based on the square of the coefficients.<a href="#cite_note-:1-4">[4]</a></li></ul>

<div class="mw-heading mw-heading3"><h3 id="Dropout">Dropout</h3></div>

In the context of neural networks, the Dropout technique repeatedly ignores random subsets of neurons during training, which simulates the training of multiple neural network architectures at once to improve generalization.<a href="#cite_note-:1-4">[4]</a>

<div class="mw-heading mw-heading2"><h2 id="Classification">Classification</h2></div>

Empirical learning of classifiers (from a finite data set) is always an <a href="Underdetermination" title="Underdetermination">underdetermined</a> problem, because it attempts to infer a function of any <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle x}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle x}</annotation>

</semantics>

</math><img src="./87f9e315fd7e2ba406057a97300593c4802b53e4.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.33ex; height:1.676ex;" alt="{\displaystyle x}" loading="lazy"> given only examples <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle x_{1},x_{2},\dots ,x_{n}}">

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle x_{1},x_{2},\dots ,x_{n}}</annotation>

</semantics>

</math><img src="./75c2d357bc1b965979bf171b5ba3bac0f68961c2.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:13.528ex; height:2.009ex;" alt="{\displaystyle x_{1},x_{2},\dots ,x_{n}}" loading="lazy">.

A regularization term (or regularizer) <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle R(f)}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle R(f)}</annotation>

</semantics>

</math><img src="./fb57f1b3ae288e52318d733f1c69d3e3637318b8.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:4.852ex; height:2.843ex;" alt="{\displaystyle R(f)}" loading="lazy"> is added to a <a href="Loss_functions_for_classification" title="Loss functions for classification">loss function</a>:

</mrow>

</munder>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \min _{f}\sum _{i=1}^{n}V(f(x_{i}),y_{i})+\lambda R(f)}</annotation>

</semantics>

</math>

where <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle V}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle V}</annotation>

</semantics>

</math><img src="./af0f6064540e84211d0ffe4dac72098adfa52845.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.787ex; height:2.176ex;" alt="{\displaystyle V}" loading="lazy"> is an underlying loss function that describes the cost of predicting <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle f(x)}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle f(x)}</annotation>

</semantics>

</math><img src="./202945cce41ecebb6f643f31d119c514bec7a074.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:4.418ex; height:2.843ex;" alt="{\displaystyle f(x)}" loading="lazy"> when the label is <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle y}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle y}</annotation>

</semantics>

</math><img src="./b8a6208ec717213d4317e666f1ae872e00620a0d.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:1.155ex; height:2.009ex;" alt="{\displaystyle y}" loading="lazy">, such as the <a href="Loss_functions_for_classification#Square_loss" title="Loss functions for classification">square loss</a> or <a href="Hinge_loss" title="Hinge loss">hinge loss</a>; and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \lambda }">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \lambda }</annotation>

</semantics>

</math><img src="./b43d0ea3c9c025af1be9128e62a18fa74bedda2a.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.355ex; height:2.176ex;" alt="{\displaystyle \lambda }" loading="lazy"> is a parameter which controls the importance of the regularization term. <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle R(f)}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle R(f)}</annotation>

</semantics>

</math><img src="./fb57f1b3ae288e52318d733f1c69d3e3637318b8.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:4.852ex; height:2.843ex;" alt="{\displaystyle R(f)}" loading="lazy"> is typically chosen to impose a penalty on the complexity of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle f}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle f}</annotation>

</semantics>

</math><img src="./132e57acb643253e7810ee9702d9581f159a1c61.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:1.279ex; height:2.509ex;" alt="{\displaystyle f}" loading="lazy">. Concrete notions of complexity used include restrictions for <a href="Smooth_function" class="mw-redirect" title="Smooth function">smoothness</a> and bounds on the <a href="Normed_vector_space" title="Normed vector space">vector space norm</a>.<a href="#cite_note-:0-5">[5]</a>

A theoretical justification for regularization is that it attempts to impose <a href="Occam's_razor" title="Occam's razor">Occam's razor</a> on the solution (as depicted in the figure above, where the green function, the simpler one, may be preferred). From a <a href="Bayesian_inference" title="Bayesian inference">Bayesian</a> point of view, many regularization techniques correspond to imposing certain <a href="Prior_probability" title="Prior probability">prior</a> distributions on model parameters.<a href="#cite_note-6">[6]</a>

Regularization can serve multiple purposes, including learning simpler models, inducing models to be sparse and introducing group structure into the learning problem.

The same idea arose in many fields of <a href="Science" title="Science">science</a>. A simple form of regularization applied to <a href="Integral_equation" title="Integral equation">integral equations</a> (<a href="Tikhonov_regularization" class="mw-redirect" title="Tikhonov regularization">Tikhonov regularization</a>) is essentially a trade-off between fitting the data and reducing a norm of the solution. More recently, non-linear regularization methods, including <a href="Total_variation_regularization" class="mw-redirect" title="Total variation regularization">total variation regularization</a>, have become popular.

<div class="mw-heading mw-heading3"><h3 id="Generalization">Generalization</h3></div>

/* start https://en.wikipedia.org/ */

.mw-parser-output .hatnote{font-style:italic}.mw-parser-output div.hatnote{padding-left:1.6em;margin-bottom:0.5em}.mw-parser-output .hatnote i{font-style:normal}.mw-parser-output .hatnote+link+.hatnote{margin-top:-0.5em}@media print{body.ns-0 .mw-parser-output .hatnote{display:none!important}}

/* end https://en.wikipedia.org/ */

</style><div role="note" class="hatnote navigation-not-searchable">Main article: <a href="Generalization_error" title="Generalization error">Generalization error</a></div>

Regularization can be motivated as a technique to improve the generalizability of a learned model.

The goal of this learning problem is to find a function that fits or predicts the outcome (label) that minimizes the expected error over all possible inputs and labels. The expected error of a function <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle f_{n}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle f_{n}}</annotation>

</semantics>

</math><img src="./b2702450f0458a5e01a698e248af552a7fab2b50.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.358ex; height:2.509ex;" alt="{\displaystyle f_{n}}" loading="lazy"> is:

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle I[f_{n}]=\int _{X\times Y}V(f_{n}(x),y)\rho (x,y)\,dx\,dy}</annotation>

</semantics>

</math>

where <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle X}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle X}</annotation>

</semantics>

</math><img src="./68baa052181f707c662844a465bfeeb135e82bab.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.98ex; height:2.176ex;" alt="{\displaystyle X}" loading="lazy"> and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle Y}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle Y}</annotation>

</semantics>

</math><img src="./961d67d6b454b4df2301ac571808a3538b3a6d3f.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.171ex; width:1.773ex; height:2.009ex;" alt="{\displaystyle Y}" loading="lazy"> are the domains of input data <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle x}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle x}</annotation>

</semantics>

</math><img src="./87f9e315fd7e2ba406057a97300593c4802b53e4.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.33ex; height:1.676ex;" alt="{\displaystyle x}" loading="lazy"> and their labels <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle y}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle y}</annotation>

</semantics>

</math><img src="./b8a6208ec717213d4317e666f1ae872e00620a0d.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:1.155ex; height:2.009ex;" alt="{\displaystyle y}" loading="lazy"> respectively.

Typically in learning problems, only a subset of input data and labels are available, measured with some noise. Therefore, the expected error is unmeasurable, and the best surrogate available is the empirical error over the <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle N}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle N}</annotation>

</semantics>

</math><img src="./f5e3890c981ae85503089652feb48b191b57aae3.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:2.064ex; height:2.176ex;" alt="{\displaystyle N}" loading="lazy"> available samples:

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<mfrac>

</mfrac>

</mrow>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

<msub>

<mover>

</mover>

</mrow>

</mrow>

</msub>

<msub>

<mover>

</mover>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle I_{S}[f_{n}]={\frac {1}{n}}\sum _{i=1}^{N}V(f_{n}({\hat {x}}_{i}),{\hat {y}}_{i})}</annotation>

</semantics>

</math>

Without bounds on the complexity of the function space (formally, the <a href="Reproducing_kernel_Hilbert_space" title="Reproducing kernel Hilbert space">reproducing kernel Hilbert space</a>) available, a model will be learned that incurs zero loss on the surrogate empirical error. If measurements (e.g. of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle x_{i}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle x_{i}}</annotation>

</semantics>

</math><img src="./e87000dd6142b81d041896a30fe58f0c3acb2158.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.129ex; height:2.009ex;" alt="{\displaystyle x_{i}}" loading="lazy">) were made with noise, this model may suffer from <a href="Overfitting" title="Overfitting">overfitting</a> and display poor expected error. Regularization introduces a penalty for exploring certain regions of the function space used to build the model, which can improve generalization.

<div class="mw-heading mw-heading2"><h2 id="Tikhonov_regularization_(ridge_regression)">Tikhonov regularization (ridge regression)</h2></div>

<div role="note" class="hatnote navigation-not-searchable">Main article: <a href="Tikhonov_regularization" class="mw-redirect" title="Tikhonov regularization">Tikhonov regularization</a></div>

These techniques are named for <a href="Andrey_Nikolayevich_Tikhonov" class="mw-redirect" title="Andrey Nikolayevich Tikhonov">Andrey Nikolayevich Tikhonov</a>, who applied regularization to <a href="Integral_equation" title="Integral equation">integral equations</a> and made important contributions in many other areas.

When learning a linear function <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle f}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle f}</annotation>

</semantics>

</math><img src="./132e57acb643253e7810ee9702d9581f159a1c61.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:1.279ex; height:2.509ex;" alt="{\displaystyle f}" loading="lazy">, characterized by an unknown <a href="Vector_space" title="Vector space">vector</a> <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle w}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle w}</annotation>

</semantics>

</math><img src="./88b1e0c8e1be5ebe69d18a8010676fa42d7961e6.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.664ex; height:1.676ex;" alt="{\displaystyle w}" loading="lazy"> such that <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle f(x)=w\cdot x}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle f(x)=w\cdot x}</annotation>

</semantics>

</math><img src="./6d5ff2299b2423508dcbb92773fd4d18b6d43dd6.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:12.189ex; height:2.843ex;" alt="{\displaystyle f(x)=w\cdot x}" loading="lazy">, one can add the <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle L_{2}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle L_{2}}</annotation>

</semantics>

</math><img src="./c6a952cfe42c86b7741f55a817da0e251793a358.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.637ex; height:2.509ex;" alt="{\displaystyle L_{2}}" loading="lazy">-norm of the vector <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle w}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle w}</annotation>

</semantics>

</math><img src="./88b1e0c8e1be5ebe69d18a8010676fa42d7961e6.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.664ex; height:1.676ex;" alt="{\displaystyle w}" loading="lazy"> to the loss expression in order to prefer solutions with smaller norms. Tikhonov regularization is one of the most common forms. It is also known as ridge regression. It is expressed as:

</mrow>

</munder>

</mrow>

</mrow>

</munderover>

<msub>

<mover>

</mover>

</mrow>

</mrow>

</msub>

<msub>

<mover>

</mover>

</mrow>

</mrow>

</msub>

<mrow>

</mrow>

</mrow>

</mrow>

</msubsup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \min _{w}\sum _{i=1}^{n}V({\hat {x}}_{i}\cdot w,{\hat {y}}_{i})+\lambda \left\|w\right\|_{2}^{2},}</annotation>

</semantics>

</math>

where <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle ({\hat {x}}_{i},{\hat {y}}_{i}),\,1\leq i\leq n,}">

<msub>

<mover>

</mover>

</mrow>

</mrow>

</msub>

<msub>

<mover>

</mover>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle ({\hat {x}}_{i},{\hat {y}}_{i}),\,1\leq i\leq n,}</annotation>

</semantics>

</math><img src="./1161d93f48a6b936ca488173be50af4a33171d01.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:18.699ex; height:2.843ex;" alt="{\displaystyle ({\hat {x}}_{i},{\hat {y}}_{i}),\,1\leq i\leq n,}" loading="lazy"> would represent samples used for training.

In the case of a general function, the norm of the function in its <a href="Reproducing_kernel_Hilbert_space" title="Reproducing kernel Hilbert space">reproducing kernel Hilbert space</a> is:

</mrow>

</munder>

</mrow>

</mrow>

</munderover>

<msub>

<mover>

</mover>

</mrow>

</mrow>

</msub>

<msub>

<mover>

</mover>

</mrow>

</mrow>

</msub>

<mrow>

</mrow>

</mrow>

</mrow>

</msubsup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \min _{f}\sum _{i=1}^{n}V(f({\hat {x}}_{i}),{\hat {y}}_{i})+\lambda \left\|f\right\|_{\mathcal {H}}^{2}}</annotation>

</semantics>

</math>

As the <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle L_{2}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle L_{2}}</annotation>

</semantics>

</math><img src="./c6a952cfe42c86b7741f55a817da0e251793a358.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.637ex; height:2.509ex;" alt="{\displaystyle L_{2}}" loading="lazy"> norm is <a href="Differentiable_function#Differentiability_in_higher_dimensions" title="Differentiable function">differentiable</a>, learning can be advanced by <a href="Gradient_descent" title="Gradient descent">gradient descent</a>.

<div class="mw-heading mw-heading3"><h3 id="Tikhonov-regularized_least_squares">Tikhonov-regularized least squares</h3></div>

The learning problem with the <a href="Least_squares" title="Least squares">least squares</a> loss function and Tikhonov regularization can be solved analytically. Written in matrix form, the optimal <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle w}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle w}</annotation>

</semantics>

</math><img src="./88b1e0c8e1be5ebe69d18a8010676fa42d7961e6.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.664ex; height:1.676ex;" alt="{\displaystyle w}" loading="lazy"> is the one for which the gradient of the loss function with respect to <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle w}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle w}</annotation>

</semantics>

</math><img src="./88b1e0c8e1be5ebe69d18a8010676fa42d7961e6.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.664ex; height:1.676ex;" alt="{\displaystyle w}" loading="lazy"> is 0.

</mrow>

</munder>

<mfrac>

</mfrac>

</mrow>

<msup>

<mrow>

<mrow>

<mover>

</mover>

</mrow>

</mrow>

</mrow>

</mrow>

</msup>

<mrow>

<mrow>

<mover>

</mover>

</mrow>

</mrow>

</mrow>

<mrow>

</mrow>

</mrow>

</mrow>

</msubsup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \min _{w}{\frac {1}{n}}\left({\hat {X}}w-Y\right)^{\mathsf {T}}\left({\hat {X}}w-Y\right)+\lambda \left\|w\right\|_{2}^{2}}</annotation>

</semantics>

</math>

<msub>

</mrow>

</msub>

<mfrac>

</mfrac>

</mrow>

<msup>

<mover>

</mover>

</mrow>

</mrow>

</msup>

<mrow>

<mrow>

<mover>

</mover>

</mrow>

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \nabla _{w}={\frac {2}{n}}{\hat {X}}^{\mathsf {T}}\left({\hat {X}}w-Y\right)+2\lambda w}</annotation>

</semantics>

</math>

<msup>

<mover>

</mover>

</mrow>

</mrow>

</msup>

<mrow>

<mrow>

<mover>

</mover>

</mrow>

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle 0={\hat {X}}^{\mathsf {T}}\left({\hat {X}}w-Y\right)+n\lambda w}</annotation>

</semantics>

</math><math display="block" xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle w=\left({\hat {X}}^{\mathsf {T}}{\hat {X}}+\lambda nI\right)^{-1}\left({\hat {X}}^{\mathsf {T}}Y\right)}">

<msup>

<mrow>

<mrow>

<msup>

<mover>

</mover>

</mrow>

</mrow>

</msup>

<mover>

</mover>

</mrow>

</mrow>

</mrow>

</mrow>

</msup>

<mrow>

<mrow>

<msup>

<mover>

</mover>

</mrow>

</mrow>

</msup>

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle w=\left({\hat {X}}^{\mathsf {T}}{\hat {X}}+\lambda nI\right)^{-1}\left({\hat {X}}^{\mathsf {T}}Y\right)}</annotation>

</semantics>

</math>

where the third statement is a <a href="First-order_condition" class="mw-redirect" title="First-order condition">first-order condition</a>.

By construction of the optimization problem, other values of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle w}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle w}</annotation>

</semantics>

</math><img src="./88b1e0c8e1be5ebe69d18a8010676fa42d7961e6.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.664ex; height:1.676ex;" alt="{\displaystyle w}" loading="lazy"> give larger values for the loss function. This can be verified by examining the <a href="Second_derivative" title="Second derivative">second derivative</a> <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \nabla _{ww}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \nabla _{ww}}</annotation>

</semantics>

</math><img src="./9df074aaacb814a1dc62001766847b1c0457c358.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:4.522ex; height:2.509ex;" alt="{\displaystyle \nabla _{ww}}" loading="lazy">.

During training, this algorithm takes <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle O(d^{3}+nd^{2})}">

<msup>

</mrow>

</msup>

<msup>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle O(d^{3}+nd^{2})}</annotation>

</semantics>

</math><img src="./fb07ee3b6722d3b0e2537d36e2a75669325c49c8.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:12.362ex; height:3.176ex;" alt="{\displaystyle O(d^{3}+nd^{2})}" loading="lazy"> <a href="Time_complexity" title="Time complexity">time</a>. The terms correspond to the matrix inversion and calculating <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle X^{\mathsf {T}}X}">

<msup>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle X^{\mathsf {T}}X}</annotation>

</semantics>

</math><img src="./75a4cb7b8bf2de8f4b677bee7ac95cd1fd3ef35f.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:5.328ex; height:2.676ex;" alt="{\displaystyle X^{\mathsf {T}}X}" loading="lazy">, respectively. Testing takes <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle O(nd)}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle O(nd)}</annotation>

</semantics>

</math><img src="./d627081a425c8784d1e2f4da283e073fd960e881.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:6.193ex; height:2.843ex;" alt="{\displaystyle O(nd)}" loading="lazy"> time.

<div class="mw-heading mw-heading2"><h2 id="Early_stopping_2">Early stopping</h2></div>

<div role="note" class="hatnote navigation-not-searchable">Main article: <a href="Early_stopping" title="Early stopping">Early stopping</a></div>

Early stopping can be viewed as regularization in time. Intuitively, a training procedure such as gradient descent tends to learn more and more complex functions with increasing iterations. By regularizing for time, model complexity can be controlled, improving generalization.

Early stopping is implemented using one data set for training, one statistically independent data set for validation and another for testing. The model is trained until performance on the validation set no longer improves and then applied to the test set.

<div class="mw-heading mw-heading3"><h3 id="Theoretical_motivation_in_least_squares">Theoretical motivation in least squares</h3></div>

Consider the finite approximation of <a href="Neumann_series" title="Neumann series">Neumann series</a> for an invertible matrix A where <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \left\|I-A\right\|<1}">

<mrow>

<mrow>

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \left\|I-A\right\|<1}</annotation>

</semantics>

</math><img src="./ab22351caf868979f543b253b84af3bc91006765.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:12.341ex; height:2.843ex;" alt="{\displaystyle \left\|I-A\right\|<1}" loading="lazy">:

</mrow>

</mrow>

</munderover>

<msup>

<mrow>

<mrow>

</mrow>

</mrow>

</mrow>

</msup>

<msup>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \sum _{i=0}^{T-1}\left(I-A\right)^{i}\approx A^{-1}}</annotation>

</semantics>

</math>

This can be used to approximate the analytical solution of unregularized least squares, if γ is introduced to ensure the norm is less than one.

<msub>

</mrow>

</msub>

<mfrac>

</mfrac>

</mrow>

</mrow>

</mrow>

</munderover>

<msup>

<mrow>

<mrow>

<mfrac>

</mfrac>

</mrow>

<msup>

<mover>

</mover>

</mrow>

</mrow>

</msup>

<mover>

</mover>

</mrow>

</mrow>

</mrow>

</msup>

<msup>

<mover>

</mover>

</mrow>

</mrow>

</msup>

<mover>

</mover>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle w_{T}={\frac {\gamma }{n}}\sum _{i=0}^{T-1}\left(I-{\frac {\gamma }{n}}{\hat {X}}^{\mathsf {T}}{\hat {X}}\right)^{i}{\hat {X}}^{\mathsf {T}}{\hat {Y}}}</annotation>

</semantics>

</math>

The exact solution to the unregularized least squares learning problem minimizes the empirical error, but may fail. By limiting T, the only free parameter in the algorithm above, the problem is regularized for time, which may improve its generalization.

The algorithm above is equivalent to restricting the number of gradient descent iterations for the empirical risk

<msub>

</mrow>

</msub>

<mfrac>

<mrow>

</mrow>

</mfrac>

</mrow>

<mrow>

<mrow>

<mover>

</mover>

</mrow>

<mover>

</mover>

</mrow>

</mrow>

<msup>

</mrow>

</mrow>

</msup>

</mrow>

</mrow>

</msubsup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle I_{s}[w]={\frac {1}{2n}}\left\|{\hat {X}}w-{\hat {Y}}\right\|_{\mathbb {R} ^{n}}^{2}}</annotation>

</semantics>

</math>

with the gradient descent update:

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

<mrow>

<mrow>

<mfrac>

</mfrac>

</mrow>

<msup>

<mover>

</mover>

</mrow>

</mrow>

</msup>

<mover>

</mover>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

<mfrac>

</mfrac>

</mrow>

<msup>

<mover>

</mover>

</mrow>

</mrow>

</msup>

<mover>

</mover>

</mrow>

</mtd>

</mtr>

</mtable>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\begin{aligned}w_{0}&=0\\[1ex]w_{t+1}&=\left(I-{\frac {\gamma }{n}}{\hat {X}}^{\mathsf {T}}{\hat {X}}\right)w_{t}+{\frac {\gamma }{n}}{\hat {X}}^{\mathsf {T}}{\hat {Y}}\end{aligned}}}</annotation>

</semantics>

</math>

The base case is trivial. The inductive case is proved as follows:

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

<mrow>

<mrow>

<mfrac>

</mfrac>

</mrow>

<msup>

<mover>

</mover>

</mrow>

</mrow>

</msup>

<mover>

</mover>

</mrow>

</mrow>

<mfrac>

</mfrac>

</mrow>

</mrow>

</mrow>

</munderover>

<msup>

<mrow>

<mrow>

<mfrac>

</mfrac>

</mrow>

<msup>

<mover>

</mover>

</mrow>

</mrow>

</msup>

<mover>

</mover>

</mrow>

</mrow>

</mrow>

</msup>

<msup>

<mover>

</mover>

</mrow>

</mrow>

</msup>

<mover>

</mover>

</mrow>

<mfrac>

</mfrac>

</mrow>

<msup>

<mover>

</mover>

</mrow>

</mrow>

</msup>

<mover>

</mover>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<mfrac>

</mfrac>

</mrow>

</mrow>

</mrow>

</munderover>

<msup>

<mrow>

<mrow>

<mfrac>

</mfrac>

</mrow>

<msup>

<mover>

</mover>

</mrow>

</mrow>

</msup>

<mover>

</mover>

</mrow>

</mrow>

</mrow>

</msup>

<msup>

<mover>

</mover>

</mrow>

</mrow>

</msup>

<mover>

</mover>

</mrow>

<mfrac>

</mfrac>

</mrow>

<msup>

<mover>

</mover>

</mrow>

</mrow>

</msup>

<mover>

</mover>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<mfrac>

</mfrac>

</mrow>

</mrow>

</mrow>

</munderover>

<msup>

<mrow>

<mrow>

<mfrac>

</mfrac>

</mrow>

<msup>

<mover>

</mover>

</mrow>

</mrow>

</msup>

<mover>

</mover>

</mrow>

</mrow>

</mrow>

</msup>

<msup>

<mover>

</mover>

</mrow>

</mrow>

</msup>

<mover>

</mover>

</mrow>

</mtd>

</mtr>

</mtable>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\begin{aligned}w_{T}&=\left(I-{\frac {\gamma }{n}}{\hat {X}}^{\mathsf {T}}{\hat {X}}\right){\frac {\gamma }{n}}\sum _{i=0}^{T-2}\left(I-{\frac {\gamma }{n}}{\hat {X}}^{\mathsf {T}}{\hat {X}}\right)^{i}{\hat {X}}^{\mathsf {T}}{\hat {Y}}+{\frac {\gamma }{n}}{\hat {X}}^{\mathsf {T}}{\hat {Y}}\\[1ex]&={\frac {\gamma }{n}}\sum _{i=1}^{T-1}\left(I-{\frac {\gamma }{n}}{\hat {X}}^{\mathsf {T}}{\hat {X}}\right)^{i}{\hat {X}}^{\mathsf {T}}{\hat {Y}}+{\frac {\gamma }{n}}{\hat {X}}^{\mathsf {T}}{\hat {Y}}\\[1ex]&={\frac {\gamma }{n}}\sum _{i=0}^{T-1}\left(I-{\frac {\gamma }{n}}{\hat {X}}^{\mathsf {T}}{\hat {X}}\right)^{i}{\hat {X}}^{\mathsf {T}}{\hat {Y}}\end{aligned}}}</annotation>

</semantics>

</math>

<div class="mw-heading mw-heading2"><h2 id="Regularizers_for_sparsity">Regularizers for sparsity</h2></div>

Assume that a dictionary <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \phi _{j}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \phi _{j}}</annotation>

</semantics>

</math><img src="./e4cbf9f77e99bf49bdc5b3afa8b6b77d15b9be6b.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:2.295ex; height:2.843ex;" alt="{\displaystyle \phi _{j}}" loading="lazy"> with dimension <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle p}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle p}</annotation>

</semantics>

</math><img src="./81eac1e205430d1f40810df36a0edffdc367af36.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; margin-left: -0.089ex; width:1.259ex; height:2.009ex;" alt="{\displaystyle p}" loading="lazy"> is given such that a function in the function space can be expressed as:

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle f(x)=\sum _{j=1}^{p}\phi _{j}(x)w_{j}}</annotation>

</semantics>

</math>

Enforcing a sparsity constraint on <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle w}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle w}</annotation>

</semantics>

</math><img src="./88b1e0c8e1be5ebe69d18a8010676fa42d7961e6.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.664ex; height:1.676ex;" alt="{\displaystyle w}" loading="lazy"> can lead to simpler and more interpretable models. This is useful in many real-life applications such as <a href="Computational_biology" title="Computational biology">computational biology</a>. An example is developing a simple predictive test for a disease in order to minimize the cost of performing medical tests while maximizing predictive power.

A sensible sparsity constraint is the <a href="Norm_(mathematics)" title="Norm (mathematics)"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle L_{0}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle L_{0}}</annotation>

</semantics>

</math><img src="./db742b8c210fc611329a4c2dcc3af4b4e1a110cb.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.637ex; height:2.509ex;" alt="{\displaystyle L_{0}}" loading="lazy"> norm</a> <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \|w\|_{0}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \|w\|_{0}}</annotation>

</semantics>

</math><img src="./8c88dffe8814e949d43dd9c949e26b1ebab06642.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:5.043ex; height:2.843ex;" alt="{\displaystyle \|w\|_{0}}" loading="lazy">, defined as the number of non-zero elements in <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle w}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle w}</annotation>

</semantics>

</math><img src="./88b1e0c8e1be5ebe69d18a8010676fa42d7961e6.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.664ex; height:1.676ex;" alt="{\displaystyle w}" loading="lazy">. Solving a <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle L_{0}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle L_{0}}</annotation>

</semantics>

</math><img src="./db742b8c210fc611329a4c2dcc3af4b4e1a110cb.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.637ex; height:2.509ex;" alt="{\displaystyle L_{0}}" loading="lazy"> regularized learning problem, however, has been demonstrated to be <a href="NP-hardness" title="NP-hardness">NP-hard</a>.<a href="#cite_note-7">[7]</a>

The <a href="Taxicab_geometry" title="Taxicab geometry"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle L_{1}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle L_{1}}</annotation>

</semantics>

</math><img src="./0e79dc1b001f8b923df475ed14de023cbc456013.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.637ex; height:2.509ex;" alt="{\displaystyle L_{1}}" loading="lazy"> norm</a> (see also <a href="Norm_(mathematics)" title="Norm (mathematics)">Norms</a>) can be used to approximate the optimal <a href="Norm_(mathematics)" title="Norm (mathematics)"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle L_{0}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle L_{0}}</annotation>

</semantics>

</math><img src="./db742b8c210fc611329a4c2dcc3af4b4e1a110cb.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.637ex; height:2.509ex;" alt="{\displaystyle L_{0}}" loading="lazy"></a> norm via convex relaxation. It can be shown that the <a href="Norm_(mathematics)" title="Norm (mathematics)"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle L_{1}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle L_{1}}</annotation>

</semantics>

</math><img src="./0e79dc1b001f8b923df475ed14de023cbc456013.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.637ex; height:2.509ex;" alt="{\displaystyle L_{1}}" loading="lazy"></a> norm induces sparsity. In the case of least squares, this problem is known as <a href="Lasso_(statistics)" title="Lasso (statistics)">LASSO</a> in statistics and <a href="Basis_pursuit" title="Basis pursuit">basis pursuit</a> in signal processing.

<msup>

</mrow>

</mrow>

</msup>

</mrow>

</munder>

<mfrac>

</mfrac>

</mrow>

<msup>

<mrow>

<mrow>

<mover>

</mover>

</mrow>

<mover>

</mover>

</mrow>

</mrow>

</mrow>

</msup>

<msub>

<mrow>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \min _{w\in \mathbb {R} ^{p}}{\frac {1}{n}}\left\|{\hat {X}}w-{\hat {Y}}\right\|^{2}+\lambda \left\|w\right\|_{1}}</annotation>

</semantics>

</math>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle L_{1}}</annotation>

</semantics>

</math><img src="./0e79dc1b001f8b923df475ed14de023cbc456013.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.637ex; height:2.509ex;" alt="{\displaystyle L_{1}}" loading="lazy"></a> regularization can occasionally produce non-unique solutions. A simple example is provided in the figure when the space of possible solutions lies on a 45 degree line. This can be problematic for certain applications, and is overcome by combining <a href="Norm_(mathematics)" title="Norm (mathematics)"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle L_{1}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle L_{1}}</annotation>

</semantics>

</math><img src="./0e79dc1b001f8b923df475ed14de023cbc456013.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.637ex; height:2.509ex;" alt="{\displaystyle L_{1}}" loading="lazy"></a> with <a href="Norm_(mathematics)" title="Norm (mathematics)"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle L_{2}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle L_{2}}</annotation>

</semantics>

</math><img src="./c6a952cfe42c86b7741f55a817da0e251793a358.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.637ex; height:2.509ex;" alt="{\displaystyle L_{2}}" loading="lazy"></a> regularization in <a href="Elastic_net_regularization" title="Elastic net regularization">elastic net regularization</a>, which takes the following form:

<msup>

</mrow>

</mrow>

</msup>

</mrow>

</munder>

<mfrac>

</mfrac>

</mrow>

<msup>

<mrow>

<mrow>

<mover>

</mover>

</mrow>

<mover>

</mover>

</mrow>

</mrow>

</mrow>

</msup>

<mrow>

<mrow>

<msub>

<mrow>

</mrow>

</mrow>

</msub>

<mrow>

</mrow>

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \min _{w\in \mathbb {R} ^{p}}{\frac {1}{n}}\left\|{\hat {X}}w-{\hat {Y}}\right\|^{2}+\lambda \left(\alpha \left\|w\right\|_{1}+(1-\alpha )\left\|w\right\|_{2}^{2}\right),\alpha \in [0,1]}</annotation>

</semantics>

</math>

Elastic net regularization tends to have a grouping effect, where correlated input features are assigned equal weights.

Elastic net regularization is commonly used in practice and is implemented in many machine learning libraries.

<div class="mw-heading mw-heading3"><h3 id="Proximal_methods">Proximal methods</h3></div>

<div role="note" class="hatnote navigation-not-searchable">Main article: <a href="Proximal_gradient_method" title="Proximal gradient method">Proximal gradient method</a></div>While the <a href="Norm_(mathematics)" title="Norm (mathematics)"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle L_{1}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle L_{1}}</annotation>

</semantics>

</math><img src="./0e79dc1b001f8b923df475ed14de023cbc456013.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.637ex; height:2.509ex;" alt="{\displaystyle L_{1}}" loading="lazy"></a> norm does not result in an NP-hard problem, the <a href="Norm_(mathematics)" title="Norm (mathematics)"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle L_{1}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle L_{1}}</annotation>

</semantics>

</math><img src="./0e79dc1b001f8b923df475ed14de023cbc456013.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.637ex; height:2.509ex;" alt="{\displaystyle L_{1}}" loading="lazy"></a> norm is convex but is not strictly differentiable due to the kink at x = 0. <a href="Subgradient_method" title="Subgradient method">Subgradient methods</a> which rely on the <a href="Subderivative" title="Subderivative">subderivative</a> can be used to solve <a href="Norm_(mathematics)" title="Norm (mathematics)"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle L_{1}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle L_{1}}</annotation>

</semantics>

</math><img src="./0e79dc1b001f8b923df475ed14de023cbc456013.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.637ex; height:2.509ex;" alt="{\displaystyle L_{1}}" loading="lazy"></a> regularized learning problems. However, faster convergence can be achieved through proximal methods.

For a problem <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \min _{w\in H}F(w)+R(w)}">

</mrow>

</munder>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \min _{w\in H}F(w)+R(w)}</annotation>

</semantics>

</math><img src="./0a5487b2c176722605917b8965873a8382bd1186.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -2.005ex; width:17.554ex; height:4.009ex;" alt="{\displaystyle \min _{w\in H}F(w)+R(w)}" loading="lazy"> such that <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle F}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle F}</annotation>

</semantics>

</math><img src="./545fd099af8541605f7ee55f08225526be88ce57.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.741ex; height:2.176ex;" alt="{\displaystyle F}" loading="lazy"> is convex, continuous, differentiable, with Lipschitz continuous gradient (such as the least squares loss function), and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle R}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle R}</annotation>

</semantics>

</math><img src="./4b0bfb3769bf24d80e15374dc37b0441e2616e33.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.764ex; height:2.176ex;" alt="{\displaystyle R}" loading="lazy"> is convex, continuous, and proper, then the proximal method to solve the problem is as follows. First define the <a href="Proximal_operator" title="Proximal operator">proximal operator</a>

<msub>

</mrow>

</msub>

<mi>argmin</mi>

</mrow>

<msup>

</mrow>

</mrow>

</msup>

</mrow>

</munder>

<mrow>

<mrow>

<mfrac>

</mfrac>

</mrow>

<msup>

<mrow>

<mrow>

</mrow>

</mrow>

</mrow>

</msup>

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \operatorname {prox} _{R}(v)=\mathop {\operatorname {argmin} } _{w\in \mathbb {R} ^{D}}\left\{R(w)+{\frac {1}{2}}\left\|w-v\right\|^{2}\right\},}</annotation>

</semantics>

</math>

and then iterate

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</munder>

<mrow>

<mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle w_{k+1}=\mathop {\operatorname {prox} } _{\gamma ,R}\left(w_{k}-\gamma \nabla F(w_{k})\right)}</annotation>

</semantics>

</math>

The proximal method iteratively performs gradient descent and then projects the result back into the space permitted by <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle R}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle R}</annotation>

</semantics>

</math><img src="./4b0bfb3769bf24d80e15374dc37b0441e2616e33.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.764ex; height:2.176ex;" alt="{\displaystyle R}" loading="lazy">.

When <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle R}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle R}</annotation>

</semantics>

</math><img src="./4b0bfb3769bf24d80e15374dc37b0441e2616e33.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.764ex; height:2.176ex;" alt="{\displaystyle R}" loading="lazy"> is the <a href="Norm_(mathematics)" title="Norm (mathematics)">L1</a> regularizer, the proximal operator is equivalent to the soft-thresholding operator,

<math display="block" xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle S_{\lambda }(v)f(n)={\begin{cases}v_{i}-\lambda ,&{\text{if }}v_{i}>\lambda \\0,&{\text{if }}v_{i}\in [-\lambda ,\lambda ]\\v_{i}+\lambda ,&{\text{if }}v_{i}<-\lambda \end{cases}}}">

<msub>

</mrow>

</msub>

<mrow>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

</mrow>

<msub>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

</mtd>

<mtd>

</mrow>

<msub>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

</mrow>

<msub>

</mrow>

</msub>

</mtd>

</mtr>

</mtable>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle S_{\lambda }(v)f(n)={\begin{cases}v_{i}-\lambda ,&{\text{if }}v_{i}>\lambda \\0,&{\text{if }}v_{i}\in [-\lambda ,\lambda ]\\v_{i}+\lambda ,&{\text{if }}v_{i}<-\lambda \end{cases}}}</annotation>

</semantics>

</math>

This allows for efficient computation.

<div class="mw-heading mw-heading3"><h3 id="Group_sparsity_without_overlaps">Group sparsity without overlaps</h3></div>

Groups of features can be regularized by a sparsity constraint, which can be useful for expressing certain prior knowledge into an optimization problem.

In the case of a linear model with non-overlapping known groups, a regularizer can be defined:

</mrow>

</mrow>

</munderover>

<msub>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle R(w)=\sum _{g=1}^{G}\left\|w_{g}\right\|_{2},}</annotation>

</semantics>

</math> where <math display="block" xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \|w_{g}\|_{2}={\sqrt {\sum _{j=1}^{|G_{g}|}\left(w_{g}^{j}\right)^{2}}}}">

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msqrt>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</munderover>

<msup>

<mrow>

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

</msup>

</msqrt>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \|w_{g}\|_{2}={\sqrt {\sum _{j=1}^{|G_{g}|}\left(w_{g}^{j}\right)^{2}}}}</annotation>

</semantics>

</math>

This can be viewed as inducing a regularizer over the <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle L_{2}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle L_{2}}</annotation>

</semantics>

</math><img src="./c6a952cfe42c86b7741f55a817da0e251793a358.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.637ex; height:2.509ex;" alt="{\displaystyle L_{2}}" loading="lazy"> norm over members of each group followed by an <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle L_{1}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle L_{1}}</annotation>

</semantics>

</math><img src="./0e79dc1b001f8b923df475ed14de023cbc456013.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.637ex; height:2.509ex;" alt="{\displaystyle L_{1}}" loading="lazy"> norm over groups.

This can be solved by the proximal method, where the proximal operator is a block-wise soft-thresholding function:

<math display="block" xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \operatorname {prox} \limits _{\lambda ,R,g}(w_{g})={\begin{cases}\left(1-{\dfrac {\lambda }{\left\|w_{g}\right\|_{2}}}\right)w_{g},&{\text{if }}\left\|w_{g}\right\|_{2}>\lambda \\[1ex]0,&{\text{if }}\|w_{g}\|_{2}\leq \lambda \end{cases}}}">

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<mrow>

<mtr>

<mtd>

<mrow>

<mrow>

<mfrac>

<msub>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</msub>

</mfrac>

</mstyle>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

</mrow>

<msub>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

</mtd>

<mtd>

</mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mtd>

</mtr>

</mtable>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \operatorname {prox} \limits _{\lambda ,R,g}(w_{g})={\begin{cases}\left(1-{\dfrac {\lambda }{\left\|w_{g}\right\|_{2}}}\right)w_{g},&{\text{if }}\left\|w_{g}\right\|_{2}>\lambda \\[1ex]0,&{\text{if }}\|w_{g}\|_{2}\leq \lambda \end{cases}}}</annotation>

</semantics>

</math>

<div class="mw-heading mw-heading3"><h3 id="Group_sparsity_with_overlaps">Group sparsity with overlaps</h3></div>

The algorithm described for group sparsity without overlaps can be applied to the case where groups do overlap, in certain situations. This will likely result in some groups with all zero elements, and other groups with some non-zero and some zero elements.

If it is desired to preserve the group structure, a new regularizer can be defined:

<mrow>

<mrow>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</munderover>

<msub>

<mover>

</mover>

</mrow>

</mrow>

</msub>

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle R(w)=\inf \left\{\sum _{g=1}^{G}\|w_{g}\|_{2}:w=\sum _{g=1}^{G}{\bar {w}}_{g}\right\}}</annotation>

</semantics>

</math>

For each <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle w_{g}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle w_{g}}</annotation>

</semantics>

</math><img src="./3e28da8e516d060d7d6cef2ef308b1000a89c806.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:2.686ex; height:2.343ex;" alt="{\displaystyle w_{g}}" loading="lazy">, <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\bar {w}}_{g}}">

<msub>

<mover>

</mover>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\bar {w}}_{g}}</annotation>

</semantics>

</math><img src="./07167fb8c0903147514bf7a72f2e736eca4cd12e.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:2.686ex; height:2.676ex;" alt="{\displaystyle {\bar {w}}_{g}}" loading="lazy"> is defined as the vector such that the restriction of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\bar {w}}_{g}}">

<msub>

<mover>

</mover>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\bar {w}}_{g}}</annotation>

</semantics>

</math><img src="./07167fb8c0903147514bf7a72f2e736eca4cd12e.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:2.686ex; height:2.676ex;" alt="{\displaystyle {\bar {w}}_{g}}" loading="lazy"> to the group <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle g}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle g}</annotation>

</semantics>

</math><img src="./d3556280e66fe2c0d0140df20935a6f057381d77.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:1.116ex; height:2.009ex;" alt="{\displaystyle g}" loading="lazy"> equals <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle w_{g}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle w_{g}}</annotation>

</semantics>

</math><img src="./3e28da8e516d060d7d6cef2ef308b1000a89c806.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:2.686ex; height:2.343ex;" alt="{\displaystyle w_{g}}" loading="lazy"> and all other entries of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\bar {w}}_{g}}">

<msub>

<mover>

</mover>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\bar {w}}_{g}}</annotation>

</semantics>

</math><img src="./07167fb8c0903147514bf7a72f2e736eca4cd12e.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:2.686ex; height:2.676ex;" alt="{\displaystyle {\bar {w}}_{g}}" loading="lazy"> are zero. The regularizer finds the optimal disintegration of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle w}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle w}</annotation>

</semantics>

</math><img src="./88b1e0c8e1be5ebe69d18a8010676fa42d7961e6.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.664ex; height:1.676ex;" alt="{\displaystyle w}" loading="lazy"> into parts. It can be viewed as duplicating all elements that exist in multiple groups. Learning problems with this regularizer can also be solved with the proximal method with a complication. The proximal operator cannot be computed in closed form, but can be effectively solved iteratively, inducing an inner iteration within the proximal method iteration.

<div class="mw-heading mw-heading2"><h2 id="Regularizers_for_semi-supervised_learning">Regularizers for semi-supervised learning</h2></div>

<div role="note" class="hatnote navigation-not-searchable">Main article: <a href="Semi-supervised_learning" class="mw-redirect" title="Semi-supervised learning">Semi-supervised learning</a></div>

When labels are more expensive to gather than input examples, semi-supervised learning can be useful. Regularizers have been designed to guide learning algorithms to learn models that respect the structure of unsupervised training samples. If a symmetric weight matrix <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle W}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle W}</annotation>

</semantics>

</math><img src="./54a9c4c547f4d6111f81946cad242b18298d70b7.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:2.435ex; height:2.176ex;" alt="{\displaystyle W}" loading="lazy"> is given, a regularizer can be defined:

</mrow>

</munder>

<msub>

</mrow>

</msub>

<msup>

<mrow>

<mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle R(f)=\sum _{i,j}w_{ij}\left(f(x_{i})-f(x_{j})\right)^{2}}</annotation>

</semantics>

</math>

If <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle W_{ij}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle W_{ij}}</annotation>

</semantics>

</math><img src="./29c09e9d719bb634d8ca5a6172b0562b945bf325.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:3.671ex; height:2.843ex;" alt="{\displaystyle W_{ij}}" loading="lazy"> encodes the result of some distance metric for points <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle x_{i}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle x_{i}}</annotation>

</semantics>

</math><img src="./e87000dd6142b81d041896a30fe58f0c3acb2158.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.129ex; height:2.009ex;" alt="{\displaystyle x_{i}}" loading="lazy"> and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle x_{j}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle x_{j}}</annotation>

</semantics>

</math><img src="./5db47cb3d2f9496205a17a6856c91c1d3d363ccd.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:2.239ex; height:2.343ex;" alt="{\displaystyle x_{j}}" loading="lazy">, it is desirable that <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle f(x_{i})\approx f(x_{j})}">

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle f(x_{i})\approx f(x_{j})}</annotation>

</semantics>

</math><img src="./b4e02bfc3366dd21618d120571b12271ad361c84.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:13.643ex; height:3.009ex;" alt="{\displaystyle f(x_{i})\approx f(x_{j})}" loading="lazy">. This regularizer captures this intuition, and is equivalent to:

<msup>

<mover>

</mover>

</mrow>

</mrow>

</msup>

<mover>

</mover>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle R(f)={\bar {f}}^{\mathsf {T}}L{\bar {f}}}</annotation>

</semantics>

</math> where <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle L=D-W}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle L=D-W}</annotation>

</semantics>

</math><img src="./14ee07ee510ca6515e6668dca08cf5a0de2c2544.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.505ex; width:11.881ex; height:2.343ex;" alt="{\displaystyle L=D-W}" loading="lazy"> is the <a href="Laplacian_matrix" title="Laplacian matrix">Laplacian matrix</a> of the graph induced by <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle W}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle W}</annotation>

</semantics>

</math><img src="./54a9c4c547f4d6111f81946cad242b18298d70b7.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:2.435ex; height:2.176ex;" alt="{\displaystyle W}" loading="lazy">.

The optimization problem <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \min _{f\in \mathbb {R} ^{m}}R(f),m=u+l}">

<msup>

</mrow>

</mrow>

</msup>

</mrow>

</munder>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \min _{f\in \mathbb {R} ^{m}}R(f),m=u+l}</annotation>

</semantics>

</math><img src="./a64f689da30e245b8bad2f51ba3a4bf191efab4e.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -2.338ex; width:20.798ex; height:4.343ex;" alt="{\displaystyle \min _{f\in \mathbb {R} ^{m}}R(f),m=u+l}" loading="lazy"> can be solved analytically if the constraint <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle f(x_{i})=y_{i}}">

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle f(x_{i})=y_{i}}</annotation>

</semantics>

</math><img src="./20ade1442f5d991818460d0ecdd4f7b053cbd91c.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:10.255ex; height:2.843ex;" alt="{\displaystyle f(x_{i})=y_{i}}" loading="lazy"> is applied for all supervised samples. The labeled part of the vector <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle f}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle f}</annotation>

</semantics>

</math><img src="./132e57acb643253e7810ee9702d9581f159a1c61.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:1.279ex; height:2.509ex;" alt="{\displaystyle f}" loading="lazy"> is therefore obvious. The unlabeled part of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle f}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle f}</annotation>

</semantics>

</math><img src="./132e57acb643253e7810ee9702d9581f159a1c61.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:1.279ex; height:2.509ex;" alt="{\displaystyle f}" loading="lazy"> is solved for by:

<msub>

</mrow>

</msub>

<msup>

</mrow>

</mrow>

</msup>

</mrow>

</munder>

<msup>

</mrow>

</msup>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</mrow>

</msup>

</mrow>

</munder>

<mrow>

<mrow>

</mrow>

</mrow>

</msubsup>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</msubsup>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</msubsup>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \min _{f_{u}\in \mathbb {R} ^{u}}f^{\mathsf {T}}Lf=\min _{f_{u}\in \mathbb {R} ^{u}}\left\{f_{u}^{\mathsf {T}}L_{uu}f_{u}+f_{l}^{\mathsf {T}}L_{lu}f_{u}+f_{u}^{\mathsf {T}}L_{ul}f_{l}\right\}}</annotation>

</semantics>

</math>

<msub>

<msub>

</mrow>

</msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \nabla _{f_{u}}=2L_{uu}f_{u}+2L_{ul}Y}</annotation>

</semantics>

</math>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</msubsup>

<mrow>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle f_{u}=L_{uu}^{\dagger }\left(L_{ul}Y\right)}</annotation>

</semantics>

</math>

The pseudo-inverse can be taken because <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle L_{ul}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle L_{ul}}</annotation>

</semantics>

</math><img src="./de9f0f868a225b3d179c9d7687a693c7488cb787.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:3.246ex; height:2.509ex;" alt="{\displaystyle L_{ul}}" loading="lazy"> has the same range as <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle L_{uu}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle L_{uu}}</annotation>

</semantics>

</math><img src="./8b8c142b8f58dbd2bd5dc50136f33cf446df5c19.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:3.696ex; height:2.509ex;" alt="{\displaystyle L_{uu}}" loading="lazy">.

<div class="mw-heading mw-heading2"><h2 id="Regularizers_for_multitask_learning">Regularizers for multitask learning</h2></div>

<div role="note" class="hatnote navigation-not-searchable">Main article: <a href="Multi-task_learning" title="Multi-task learning">Multi-task learning</a></div>In the case of multitask learning, <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle T}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle T}</annotation>

</semantics>

</math><img src="./ec7200acd984a1d3a3d7dc455e262fbe54f7f6e0.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.636ex; height:2.176ex;" alt="{\displaystyle T}" loading="lazy"> problems are considered simultaneously, each related in some way. The goal is to learn <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle T}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle T}</annotation>

</semantics>

</math><img src="./ec7200acd984a1d3a3d7dc455e262fbe54f7f6e0.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.636ex; height:2.176ex;" alt="{\displaystyle T}" loading="lazy"> functions, ideally borrowing strength from the relatedness of tasks, that have predictive power. This is equivalent to learning the matrix <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle W:T\times D}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle W:T\times D}</annotation>

</semantics>

</math><img src="./edf8322cb32156be2fe4daec37105dbe1a1cb1d3.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:10.773ex; height:2.176ex;" alt="{\displaystyle W:T\times D}" loading="lazy"> .

<div class="mw-heading mw-heading3"><h3 id="Sparse_regularizer_on_columns">Sparse regularizer on columns</h3></div>

</mrow>

</mrow>

</munderover>

<msub>

<mrow>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle R(w)=\sum _{i=1}^{D}\left\|W\right\|_{2,1}}</annotation>

</semantics>

</math>

This regularizer defines an L2 norm on each column and an L1 norm over all columns. It can be solved by proximal methods.

<div class="mw-heading mw-heading3"><h3 id="Nuclear_norm_regularization">Nuclear norm regularization</h3></div>

<msub>

<mrow>

<mrow>

</mrow>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle R(w)=\left\|\sigma (W)\right\|_{1}}</annotation>

</semantics>

</math> where <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \sigma (W)}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \sigma (W)}</annotation>

</semantics>

</math><img src="./f89efda565839c985b5320de98bd98b88c1cfd90.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:5.574ex; height:2.843ex;" alt="{\displaystyle \sigma (W)}" loading="lazy"> is the <a href="Eigenvalues_and_eigenvectors" title="Eigenvalues and eigenvectors">eigenvalues</a> in the <a href="Singular_value_decomposition" title="Singular value decomposition">singular value decomposition</a> of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle W}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle W}</annotation>

</semantics>

</math><img src="./54a9c4c547f4d6111f81946cad242b18298d70b7.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:2.435ex; height:2.176ex;" alt="{\displaystyle W}" loading="lazy">.

<div class="mw-heading mw-heading3"><h3 id="Mean-constrained_regularization">Mean-constrained regularization</h3></div>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</munderover>

<mrow>

<mrow>

<msub>

</mrow>

</msub>

<mfrac>

</mfrac>

</mrow>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</msubsup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle R(f_{1}\cdots f_{T})=\sum _{t=1}^{T}\left\|f_{t}-{\frac {1}{T}}\sum _{s=1}^{T}f_{s}\right\|_{H_{k}}^{2}}</annotation>

</semantics>

</math>

This regularizer constrains the functions learned for each task to be similar to the overall average of the functions across all tasks. This is useful for expressing prior information that each task is expected to share with each other task. An example is predicting blood iron levels measured at different times of the day, where each task represents an individual.

<div class="mw-heading mw-heading3"><h3 id="Clustered_mean-constrained_regularization">Clustered mean-constrained regularization</h3></div>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</munderover>

</mrow>

</munder>

<mrow>

<mrow>

<msub>

</mrow>

</msub>

<mfrac>

<mrow>

</mrow>

</mfrac>

</mrow>

</mrow>

</munder>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</msubsup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle R(f_{1}\cdots f_{T})=\sum _{r=1}^{C}\sum _{t\in I(r)}\left\|f_{t}-{\frac {1}{I(r)}}\sum _{s\in I(r)}f_{s}\right\|_{H_{k}}^{2}}</annotation>

</semantics>

</math> where <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle I(r)}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle I(r)}</annotation>

</semantics>

</math><img src="./43eabce3279a8b35ce65904e1bc775fb78e931f5.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:4.03ex; height:2.843ex;" alt="{\displaystyle I(r)}" loading="lazy"> is a cluster of tasks.

This regularizer is similar to the mean-constrained regularizer, but instead enforces similarity between tasks within the same cluster. This can capture more complex prior information. This technique has been used to predict <a href="Netflix" title="Netflix">Netflix</a> recommendations. A cluster would correspond to a group of people who share similar preferences.

<div class="mw-heading mw-heading3"><h3 id="Graph-based_similarity">Graph-based similarity</h3></div>

More generally than above, similarity between tasks can be defined by a function. The regularizer encourages the model to learn similar functions for similar tasks.

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</munderover>

<msup>

<mrow>

<mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</mrow>

</msup>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle R(f_{1}\cdots f_{T})=\sum _{t,s=1,t\neq s}^{\mathsf {T}}\left\|f_{t}-f_{s}\right\|^{2}M_{ts}}</annotation>

</semantics>

</math> for a given symmetric <a href="Similarity_matrix" class="mw-redirect" title="Similarity matrix">similarity matrix</a> <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle M}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle M}</annotation>

</semantics>

</math><img src="./f82cade9898ced02fdd08712e5f0c0151758a0dd.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:2.442ex; height:2.176ex;" alt="{\displaystyle M}" loading="lazy">.

<div class="mw-heading mw-heading2"><h2 id="Other_uses_of_regularization_in_statistics_and_machine_learning">Other uses of regularization in statistics and machine learning</h2></div>

<a href="Bayesian_model_comparison" class="mw-redirect" title="Bayesian model comparison">Bayesian learning</a> methods make use of a <a href="Prior_probability" title="Prior probability">prior probability</a> that (usually) gives lower probability to more complex models. Well-known model selection techniques include the <a href="Akaike_information_criterion" title="Akaike information criterion">Akaike information criterion</a> (AIC), <a href="Minimum_description_length" title="Minimum description length">minimum description length</a> (MDL), and the <a href="Bayesian_information_criterion" title="Bayesian information criterion">Bayesian information criterion</a> (BIC). Alternative methods of controlling overfitting not involving regularization include <a href="Cross-validation_(statistics)" title="Cross-validation (statistics)">cross-validation</a>.

Examples of applications of different methods of regularization to the <a href="Linear_model" title="Linear model">linear model</a> are:

<th>Model</th>

<th>Fit measure</th>

<th>Entropy measure<a href="#cite_note-:0-5">[5]</a><a href="#cite_note-8">[8]</a>

</th></tr>

<tr>

<msub>

<mrow>

<mrow>

</mrow>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \left\|Y-X\beta \right\|_{2}}</annotation>

</semantics>

</math><img src="./b39c6f2046cf31aeb910742324180509ec20cd91.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:11.305ex; height:3.009ex;" alt="{\displaystyle \left\|Y-X\beta \right\|_{2}}" loading="lazy"></td>

<msub>

<mrow>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \left\|\beta \right\|_{0}}</annotation>

</semantics>

</math><img src="./78ced2975d43b72c284acff5541c72285eb35c4d.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:4.711ex; height:3.009ex;" alt="{\displaystyle \left\|\beta \right\|_{0}}" loading="lazy">

</td></tr>

<tr>

<td><a href="Lasso_(statistics)" title="Lasso (statistics)">Lasso</a><a href="#cite_note-9">[9]</a>

</td>

<msub>

<mrow>

<mrow>

</mrow>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \left\|Y-X\beta \right\|_{2}}</annotation>

</semantics>

</math><img src="./b39c6f2046cf31aeb910742324180509ec20cd91.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:11.305ex; height:3.009ex;" alt="{\displaystyle \left\|Y-X\beta \right\|_{2}}" loading="lazy"></td>

<msub>

<mrow>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \left\|\beta \right\|_{1}}</annotation>

</semantics>

</math><img src="./a2764243df455528c7d6a3bc5ed74b5db7a03048.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:4.711ex; height:3.009ex;" alt="{\displaystyle \left\|\beta \right\|_{1}}" loading="lazy">

</td></tr>

<tr>

<td><a href="Ridge_regression" title="Ridge regression">Ridge regression</a><a href="#cite_note-ridge-10">[10]</a></td>

<msub>

<mrow>

<mrow>

</mrow>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \left\|Y-X\beta \right\|_{2}}</annotation>

</semantics>

</math><img src="./b39c6f2046cf31aeb910742324180509ec20cd91.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:11.305ex; height:3.009ex;" alt="{\displaystyle \left\|Y-X\beta \right\|_{2}}" loading="lazy"></td>

<msub>

<mrow>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \left\|\beta \right\|_{2}}</annotation>

</semantics>

</math><img src="./8720b0754f715ac28c8a0f674a89bcef3cb4ab6d.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:4.711ex; height:3.009ex;" alt="{\displaystyle \left\|\beta \right\|_{2}}" loading="lazy">

</td></tr>

<tr>

<td><a href="Basis_pursuit_denoising" title="Basis pursuit denoising">Basis pursuit denoising</a></td>

<msub>

<mrow>

<mrow>

</mrow>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \left\|Y-X\beta \right\|_{2}}</annotation>

</semantics>

</math><img src="./b39c6f2046cf31aeb910742324180509ec20cd91.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:11.305ex; height:3.009ex;" alt="{\displaystyle \left\|Y-X\beta \right\|_{2}}" loading="lazy"></td>

<msub>

<mrow>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \lambda \left\|\beta \right\|_{1}}</annotation>

</semantics>

</math><img src="./a7cedcbcb5706a36edaff8660a044d66305c4946.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:6.066ex; height:3.009ex;" alt="{\displaystyle \lambda \left\|\beta \right\|_{1}}" loading="lazy">

</td></tr>

<tr>

<td>Rudin–Osher–Fatemi model (TV)</td>

<msub>

<mrow>

<mrow>

</mrow>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \left\|Y-X\beta \right\|_{2}}</annotation>

</semantics>

</math><img src="./b39c6f2046cf31aeb910742324180509ec20cd91.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:11.305ex; height:3.009ex;" alt="{\displaystyle \left\|Y-X\beta \right\|_{2}}" loading="lazy"></td>

<msub>

<mrow>

<mrow>

</mrow>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \lambda \left\|\nabla \beta \right\|_{1}}</annotation>

</semantics>

</math><img src="./bcd34be35f7cb17470b1a7d7e8452a460ba3075f.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:8.002ex; height:3.009ex;" alt="{\displaystyle \lambda \left\|\nabla \beta \right\|_{1}}" loading="lazy">

</td></tr>

<tr>

<td><a href="Potts_model" title="Potts model">Potts model</a></td>

<msub>

<mrow>

<mrow>

</mrow>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \left\|Y-X\beta \right\|_{2}}</annotation>

</semantics>

</math><img src="./b39c6f2046cf31aeb910742324180509ec20cd91.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:11.305ex; height:3.009ex;" alt="{\displaystyle \left\|Y-X\beta \right\|_{2}}" loading="lazy"></td>

<msub>

<mrow>

<mrow>

</mrow>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \lambda \left\|\nabla \beta \right\|_{0}}</annotation>

</semantics>

</math><img src="./daa6cd43ea728533b40bda953107cbd154f61fa8.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:8.002ex; height:3.009ex;" alt="{\displaystyle \lambda \left\|\nabla \beta \right\|_{0}}" loading="lazy">

</td></tr>

<tr>

</td>

<msub>

<mrow>

<mrow>

</mrow>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \left\|Y-X\beta \right\|_{1}}</annotation>

</semantics>

</math><img src="./1ee70963c9118307e927548378b2f156ee913946.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:11.305ex; height:3.009ex;" alt="{\displaystyle \left\|Y-X\beta \right\|_{1}}" loading="lazy"></td>

<msub>

<mrow>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \left\|\beta \right\|_{1}}</annotation>

</semantics>

</math><img src="./a2764243df455528c7d6a3bc5ed74b5db7a03048.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:4.711ex; height:3.009ex;" alt="{\displaystyle \left\|\beta \right\|_{1}}" loading="lazy">

</td></tr>

<tr>

<td>Dantzig Selector<a href="#cite_note-12">[12]</a>

</td>

<msub>

<mrow>

<mrow>

<msup>

</mrow>

</msup>

</mrow>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \left\|X^{\mathsf {T}}(Y-X\beta )\right\|_{\infty }}</annotation>

</semantics>

</math><img src="./fde7e85fcc315f9b132818fad53c703971072329.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.338ex; width:17.284ex; height:3.676ex;" alt="{\displaystyle \left\|X^{\mathsf {T}}(Y-X\beta )\right\|_{\infty }}" loading="lazy"></td>

<msub>

<mrow>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \left\|\beta \right\|_{1}}</annotation>

</semantics>

</math><img src="./a2764243df455528c7d6a3bc5ed74b5db7a03048.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:4.711ex; height:3.009ex;" alt="{\displaystyle \left\|\beta \right\|_{1}}" loading="lazy">

</td></tr>

<tr>

<td>SLOPE<a href="#cite_note-13">[13]</a>

</td>

<msub>

<mrow>

<mrow>

</mrow>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \left\|Y-X\beta \right\|_{2}}</annotation>

</semantics>

</math><img src="./b39c6f2046cf31aeb910742324180509ec20cd91.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:11.305ex; height:3.009ex;" alt="{\displaystyle \left\|Y-X\beta \right\|_{2}}" loading="lazy"></td>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

<msub>

<mrow>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \sum _{i=1}^{p}\lambda _{i}\left|\beta \right|_{(i)}}</annotation>

</semantics>

</math><img src="./4fb309207b13b41463728134e6f22dd904088774.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -3.005ex; width:10.602ex; height:7.009ex;" alt="{\displaystyle \sum _{i=1}^{p}\lambda _{i}\left|\beta \right|_{(i)}}" loading="lazy">

</td></tr></tbody></table>

<ul><li><a href="Bayesian_interpretation_of_regularization" class="mw-redirect" title="Bayesian interpretation of regularization">Bayesian interpretation of regularization</a></li>

<li><a href="Bias%E2%80%93variance_tradeoff" title="Bias–variance tradeoff">Bias–variance tradeoff</a></li>

<li><a href="Matrix_regularization" title="Matrix regularization">Matrix regularization</a></li>

<li><a href="Regularization_by_spectral_filtering" title="Regularization by spectral filtering">Regularization by spectral filtering</a></li>

<li><a href="Regularized_least_squares" title="Regularized least squares">Regularized least squares</a></li>

<li><a href="Lagrange_multiplier" title="Lagrange multiplier">Lagrange multiplier</a></li>

<li><a href="Variance_reduction" title="Variance reduction">Variance reduction</a></li>

<li><a href="L-curve" title="L-curve">L-curve</a></li></ul>

<div class="mw-heading mw-heading2"><h2 id="Notes">Notes</h2></div>

/* start https://en.wikipedia.org/ */

.mw-parser-output .reflist{margin-bottom:0.5em;list-style-type:decimal}@media screen{.mw-parser-output .reflist{font-size:90%}}.mw-parser-output .reflist .references{font-size:100%;margin-bottom:0;list-style-type:inherit}.mw-parser-output .reflist-columns-2{column-width:30em}.mw-parser-output .reflist-columns-3{column-width:25em}.mw-parser-output .reflist-columns{margin-top:0.3em}.mw-parser-output .reflist-columns ol{margin-top:0}.mw-parser-output .reflist-columns li{page-break-inside:avoid;break-inside:avoid-column}.mw-parser-output .reflist-upper-alpha{list-style-type:upper-alpha}.mw-parser-output .reflist-upper-roman{list-style-type:upper-roman}.mw-parser-output .reflist-lower-alpha{list-style-type:lower-alpha}.mw-parser-output .reflist-lower-greek{list-style-type:lower-greek}.mw-parser-output .reflist-lower-roman{list-style-type:lower-roman}

/* end https://en.wikipedia.org/ */

</style><div class="reflist">

/* start https://en.wikipedia.org/ */

.mw-parser-output cite.citation{font-style:inherit;word-wrap:break-word}.mw-parser-output .citation q{quotes:"\"""\"""'""'"}.mw-parser-output .citation:target{background-color:rgba(0,127,255,0.133)}.mw-parser-output .id-lock-free.id-lock-free a{background:url("./mw/Lock-green.svg")right 0.1em center/9px no-repeat}.mw-parser-output .id-lock-limited.id-lock-limited a,.mw-parser-output .id-lock-registration.id-lock-registration a{background:url("./mw/Lock-gray-alt-2.svg")right 0.1em center/9px no-repeat}.mw-parser-output .id-lock-subscription.id-lock-subscription a{background:url("./mw/Lock-red-alt-2.svg")right 0.1em center/9px no-repeat}.mw-parser-output .cs1-ws-icon a{background:url("./mw/Wikisource-logo.svg")right 0.1em center/12px no-repeat}body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-free a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-limited a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-registration a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-subscription a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .cs1-ws-icon a{background-size:contain;padding:0 1em 0 0}.mw-parser-output .cs1-code{color:inherit;background:inherit;border:none;padding:inherit}.mw-parser-output .cs1-hidden-error{display:none;color:var(--color-error,#d33)}.mw-parser-output .cs1-visible-error{color:var(--color-error,#d33)}.mw-parser-output .cs1-maint{display:none;color:#085;margin-left:0.3em}.mw-parser-output .cs1-kern-left{padding-left:0.2em}.mw-parser-output .cs1-kern-right{padding-right:0.2em}.mw-parser-output .citation .mw-selflink{font-weight:inherit}@media screen{.mw-parser-output .cs1-format{font-size:95%}html.skin-theme-clientpref-night .mw-parser-output .cs1-maint{color:#18911f}}@media screen and (prefers-color-scheme:dark){html.skin-theme-clientpref-os .mw-parser-output .cs1-maint{color:#18911f}}

/* end https://en.wikipedia.org/ */

</style><cite id="CITEREFKratsios2020" class="citation journal cs1">Kratsios, Anastasis (2020). <a rel="nofollow" class="external text" href="https://doi.org/10.3390%2Frisks8020040">"Deep Arbitrage-Free Learning in a Generalized HJM Framework via Arbitrage-Regularization Data"</a>. Risks. 8 (2): <a rel="nofollow" class="external autonumber" href="https://www.mdpi.com/2227-9091/8/2/40">[1]</a>. <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/1710.05114">1710.05114</a>. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.3390%2Frisks8020040">10.3390/risks8020040</a>. <a href="Hdl_(identifier)" class="mw-redirect" title="Hdl (identifier)">hdl</a>:<a rel="nofollow" class="external text" href="https://hdl.handle.net/20.500.11850%2F456375">20.500.11850/456375</a>. <q>Term structure models can be regularized to remove arbitrage opportunities [<a href="Sic" title="Sic">sic</a>?].</q></cite>

</li>

<li id="cite_note-2"><a href="#cite_ref-2">^</a> <cite id="CITEREFBühlmannVan_De_Geer2011" class="citation book cs1">Bühlmann, Peter; Van De Geer, Sara (2011). <a rel="nofollow" class="external text" href="https://archive.org/details/statisticsforhig00bhlm">Statistics for High-Dimensional Data</a>. Springer Series in Statistics. p. <a rel="nofollow" class="external text" href="https://archive.org/details/statisticsforhig00bhlm/page/n27">9</a>. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1007%2F978-3-642-20192-9">10.1007/978-3-642-20192-9</a>. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-3-642-20191-2</bdi>. <q>If p > n, the ordinary least squares estimator is not unique and will heavily overfit the data. Thus, a form of complexity regularization will be necessary.</q></cite>

</li>

<li id="cite_note-3"><a href="#cite_ref-3">^</a> <cite id="CITEREFGoodfellowBengioCourville" class="citation book cs1">Goodfellow, Ian; Bengio, Yoshua; Courville, Aaron. <a rel="nofollow" class="external text" href="https://www.deeplearningbook.org/contents/ml.html">Deep Learning Book</a>. Retrieved 2021-01-29.</cite>

</li>

<li id="cite_note-:1-4">^ <a href="#cite_ref-:1_4-0">a</a> <a href="#cite_ref-:1_4-1">b</a> <a href="#cite_ref-:1_4-2">c</a> <a href="#cite_ref-:1_4-3">d</a> <cite id="CITEREFGuo" class="citation web cs1">Guo, Jingru. <a rel="nofollow" class="external text" href="https://deeplearning.ai/ai-notes/regularization/">"AI Notes: Regularizing neural networks"</a>. deeplearning.ai. Retrieved 2024-02-04.</cite>

</li>

<li id="cite_note-:0-5">^ <a href="#cite_ref-:0_5-0">a</a> <a href="#cite_ref-:0_5-1">b</a> <cite id="CITEREFBishop2007" class="citation book cs1">Bishop, Christopher M. (2007). Pattern recognition and machine learning (Corr. printing. ed.). New York: Springer. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-0-387-31073-2</bdi>.</cite>

</li>

<li id="cite_note-6"><a href="#cite_ref-6">^</a> For the connection between <a href="Maximum_a_posteriori_estimation" title="Maximum a posteriori estimation">maximum a posteriori estimation</a> and <a href="Ridge_regression" title="Ridge regression">ridge regression</a>, see <cite id="CITEREFWeinberger2018" class="citation web cs1">Weinberger, Kilian (July 11, 2018). <a rel="nofollow" class="external text" href="https://www.cs.cornell.edu/courses/cs4780/2018fa/lectures/lecturenote08.html#map-estimate">"Linear / Ridge Regression"</a>. CS4780 Machine Learning Lecture 13. Cornell.</cite>

</li>

<li id="cite_note-7"><a href="#cite_ref-7">^</a> <cite id="CITEREFNatarajan1995" class="citation journal cs1">Natarajan, B. (1995-04-01). <a rel="nofollow" class="external text" href="http://epubs.siam.org/doi/abs/10.1137/S0097539792240406">"Sparse Approximate Solutions to Linear Systems"</a>. SIAM Journal on Computing. 24 (2): 227–234. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1137%2FS0097539792240406">10.1137/S0097539792240406</a>. <a href="ISSN_(identifier)" class="mw-redirect" title="ISSN (identifier)">ISSN</a> <a rel="nofollow" class="external text" href="https://search.worldcat.org/issn/0097-5397">0097-5397</a>. <a href="S2CID_(identifier)" class="mw-redirect" title="S2CID (identifier)">S2CID</a> <a rel="nofollow" class="external text" href="https://api.semanticscholar.org/CorpusID:2072045">2072045</a>.</cite>

</li>

<li id="cite_note-8"><a href="#cite_ref-8">^</a> <cite id="CITEREFDuda2004" class="citation book cs1">Duda, Richard O. (2004). Pattern classification + computer manual : hardcover set (2 ed.). New York [u.a.]: Wiley. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-0-471-70350-1</bdi>.</cite>

</li>

<li id="cite_note-9"><a href="#cite_ref-9">^</a> <cite id="CITEREFTibshirani1996" class="citation journal cs1">Tibshirani, Robert (1996). <a rel="nofollow" class="external text" href="http://www-stat.stanford.edu/~tibs/ftp/lasso.ps">"Regression Shrinkage and Selection via the Lasso"</a> (<a href="PostScript" title="PostScript">PostScript</a>). <a href="Journal_of_the_Royal_Statistical_Society%2C_Series_B" class="mw-redirect" title="Journal of the Royal Statistical Society, Series B">Journal of the Royal Statistical Society, Series B</a>. 58 (1): 267–288. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1111%2Fj.2517-6161.1996.tb02080.x">10.1111/j.2517-6161.1996.tb02080.x</a>. <a href="MR_(identifier)" class="mw-redirect" title="MR (identifier)">MR</a> <a rel="nofollow" class="external text" href="https://mathscinet.ams.org/mathscinet-getitem?mr=1379242">1379242</a>. Retrieved 2009-03-19.</cite>

</li>

<li id="cite_note-ridge-10"><a href="#cite_ref-ridge_10-0">^</a> <cite id="CITEREFArthur_E._HoerlRobert_W._Kennard1970" class="citation journal cs1">Arthur E. Hoerl; Robert W. Kennard (1970). "Ridge regression: Biased estimation for nonorthogonal problems". Technometrics. 12 (1): 55–67. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.2307%2F1267351">10.2307/1267351</a>. <a href="JSTOR_(identifier)" class="mw-redirect" title="JSTOR (identifier)">JSTOR</a> <a rel="nofollow" class="external text" href="https://www.jstor.org/stable/1267351">1267351</a>.</cite>

</li>

<li id="cite_note-11"><a href="#cite_ref-11">^</a> <cite id="CITEREFLi_WangMichael_D._GordonJi_Zhu2006" class="citation conference cs1">Li Wang; Michael D. Gordon; Ji Zhu (2006). "Regularized Least Absolute Deviations Regression and an Efficient Algorithm for Parameter Tuning". Sixth International Conference on Data Mining. pp. 690–700. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1109%2FICDM.2006.134">10.1109/ICDM.2006.134</a>. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-0-7695-2701-7</bdi>.</cite>

</li>

<li id="cite_note-12"><a href="#cite_ref-12">^</a> <cite id="CITEREFCandesTao2007" class="citation journal cs1"><a href="Emmanuel_Cand%C3%A8s" title="Emmanuel Candès">Candes, Emmanuel</a>; <a href="Terence_Tao" title="Terence Tao">Tao, Terence</a> (2007). "The Dantzig selector: Statistical estimation when p is much larger than n". Annals of Statistics. 35 (6): 2313–2351. <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/math/0506081">math/0506081</a>. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1214%2F009053606000001523">10.1214/009053606000001523</a>. <a href="MR_(identifier)" class="mw-redirect" title="MR (identifier)">MR</a> <a rel="nofollow" class="external text" href="https://mathscinet.ams.org/mathscinet-getitem?mr=2382644">2382644</a>. <a href="S2CID_(identifier)" class="mw-redirect" title="S2CID (identifier)">S2CID</a> <a rel="nofollow" class="external text" href="https://api.semanticscholar.org/CorpusID:88524200">88524200</a>.</cite>

</li>

<cite id="CITEREFMałgorzata_BogdanEwout_van_den_BergWeijie_SuEmmanuel_J._Candes2013" class="citation arxiv cs1"><a href="Ma%C5%82gorzata_Bogdan" title="Małgorzata Bogdan">Małgorzata Bogdan</a>; Ewout van den Berg; Weijie Su; Emmanuel J. Candes (2013). "Statistical estimation and testing via the ordered L1 norm". <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/1310.1969">1310.1969</a> [<a rel="nofollow" class="external text" href="https://arxiv.org/archive/stat.ME">stat.ME</a>].</cite>

</li>

</ol></div></div>

<div class="mw-heading mw-heading2"><h2 id="References">References</h2></div>

<ul><li><cite id="CITEREFNeumaier1998" class="citation journal cs1">Neumaier, A. (1998). <a rel="nofollow" class="external text" href="https://web.archive.org/web/20070630163742/https://www.mat.univie.ac.at/~neum/ms/regtutorial.pdf">"Solving ill-conditioned and singular linear systems: A tutorial on regularization"</a> (PDF). SIAM Review. 40 (3): 636–666. <a href="Bibcode_(identifier)" class="mw-redirect" title="Bibcode (identifier)">Bibcode</a>:<a rel="nofollow" class="external text" href="https://ui.adsabs.harvard.edu/abs/1998SIAMR..40..636N">1998SIAMR..40..636N</a>. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1137%2FS0036144597321909">10.1137/S0036144597321909</a>. Archived from <a rel="nofollow" class="external text" href="https://www.mat.univie.ac.at/~neum/ms/regtutorial.pdf">the original</a> (PDF) on 2007-06-30.</cite></li>

<li><cite id="CITEREFKukačkaGolkovCremers2017" class="citation arxiv cs1">Kukačka, Jan; Golkov, Vladimir; Cremers, Daniel (2017). "Regularization for Deep Learning: A Taxonomy". <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/1710.10686">1710.10686</a> [<a rel="nofollow" class="external text" href="https://arxiv.org/archive/cs.LG">cs.LG</a>].</cite></li></ul>

/* start https://en.wikipedia.org/ */

.mw-parser-output .hlist dl,.mw-parser-output .hlist ol,.mw-parser-output .hlist ul{margin:0;padding:0}.mw-parser-output .hlist dd,.mw-parser-output .hlist dt,.mw-parser-output .hlist li{margin:0;display:inline}.mw-parser-output .hlist.inline,.mw-parser-output .hlist.inline dl,.mw-parser-output .hlist.inline ol,.mw-parser-output .hlist.inline ul,.mw-parser-output .hlist dl dl,.mw-parser-output .hlist dl ol,.mw-parser-output .hlist dl ul,.mw-parser-output .hlist ol dl,.mw-parser-output .hlist ol ol,.mw-parser-output .hlist ol ul,.mw-parser-output .hlist ul dl,.mw-parser-output .hlist ul ol,.mw-parser-output .hlist ul ul{display:inline}.mw-parser-output .hlist .mw-empty-li{display:none}.mw-parser-output .hlist dt::after{content:": "}.mw-parser-output .hlist dd::after,.mw-parser-output .hlist li::after{content:" · ";font-weight:bold}.mw-parser-output .hlist dd:last-child::after,.mw-parser-output .hlist dt:last-child::after,.mw-parser-output .hlist li:last-child::after{content:none}.mw-parser-output .hlist dd dd:first-child::before,.mw-parser-output .hlist dd dt:first-child::before,.mw-parser-output .hlist dd li:first-child::before,.mw-parser-output .hlist dt dd:first-child::before,.mw-parser-output .hlist dt dt:first-child::before,.mw-parser-output .hlist dt li:first-child::before,.mw-parser-output .hlist li dd:first-child::before,.mw-parser-output .hlist li dt:first-child::before,.mw-parser-output .hlist li li:first-child::before{content:" (";font-weight:normal}.mw-parser-output .hlist dd dd:last-child::after,.mw-parser-output .hlist dd dt:last-child::after,.mw-parser-output .hlist dd li:last-child::after,.mw-parser-output .hlist dt dd:last-child::after,.mw-parser-output .hlist dt dt:last-child::after,.mw-parser-output .hlist dt li:last-child::after,.mw-parser-output .hlist li dd:last-child::after,.mw-parser-output .hlist li dt:last-child::after,.mw-parser-output .hlist li li:last-child::after{content:")";font-weight:normal}.mw-parser-output .hlist ol{counter-reset:listitem}.mw-parser-output .hlist ol>li{counter-increment:listitem}.mw-parser-output .hlist ol>li::before{content:" "counter(listitem)"\a0 "}.mw-parser-output .hlist dd ol>li:first-child::before,.mw-parser-output .hlist dt ol>li:first-child::before,.mw-parser-output .hlist li ol>li:first-child::before{content:" ("counter(listitem)"\a0 "}

/* end https://en.wikipedia.org/ */

</style><style data-mw-deduplicate="TemplateStyles:r1236075235">

/* start https://en.wikipedia.org/ */

.mw-parser-output .navbox{box-sizing:border-box;border:1px solid #a2a9b1;width:100%;clear:both;font-size:88%;text-align:center;padding:1px;margin:1em auto 0}.mw-parser-output .navbox .navbox{margin-top:0}.mw-parser-output .navbox+.navbox,.mw-parser-output .navbox+.navbox-styles+.navbox{margin-top:-1px}.mw-parser-output .navbox-inner,.mw-parser-output .navbox-subgroup{width:100%}.mw-parser-output .navbox-group,.mw-parser-output .navbox-title,.mw-parser-output .navbox-abovebelow{padding:0.25em 1em;line-height:1.5em;text-align:center}.mw-parser-output .navbox-group{white-space:nowrap;text-align:right}.mw-parser-output .navbox,.mw-parser-output .navbox-subgroup{background-color:#fdfdfd}.mw-parser-output .navbox-list{line-height:1.5em;border-color:#fdfdfd}.mw-parser-output .navbox-list-with-group{text-align:left;border-left-width:2px;border-left-style:solid}.mw-parser-output tr+tr>.navbox-abovebelow,.mw-parser-output tr+tr>.navbox-group,.mw-parser-output tr+tr>.navbox-image,.mw-parser-output tr+tr>.navbox-list{border-top:2px solid #fdfdfd}.mw-parser-output .navbox-title{background-color:#ccf}.mw-parser-output .navbox-abovebelow,.mw-parser-output .navbox-group,.mw-parser-output .navbox-subgroup .navbox-title{background-color:#ddf}.mw-parser-output .navbox-subgroup .navbox-group,.mw-parser-output .navbox-subgroup .navbox-abovebelow{background-color:#e6e6ff}.mw-parser-output .navbox-even{background-color:#f7f7f7}.mw-parser-output .navbox-odd{background-color:transparent}.mw-parser-output .navbox .hlist td dl,.mw-parser-output .navbox .hlist td ol,.mw-parser-output .navbox .hlist td ul,.mw-parser-output .navbox td.hlist dl,.mw-parser-output .navbox td.hlist ol,.mw-parser-output .navbox td.hlist ul{padding:0.125em 0}.mw-parser-output .navbox .navbar{display:block;font-size:100%}.mw-parser-output .navbox-title .navbar{float:left;text-align:left;margin-right:0.5em}body.skin--responsive .mw-parser-output .navbox-image img{max-width:none!important}@media print{body.ns-0 .mw-parser-output .navbox{display:none!important}}

/* end https://en.wikipedia.org/ */

</style></div><div role="navigation" class="navbox" aria-labelledby="Artificial_intelligence_(AI)426" style="padding:3px"><table class="nowraplinks hlist mw-collapsible autocollapse navbox-inner" style="border-spacing:0;background:transparent;color:inherit"><tbody><tr><th scope="col" class="navbox-title" colspan="2"><style data-mw-deduplicate="TemplateStyles:r1239400231">

/* start https://en.wikipedia.org/ */

.mw-parser-output .navbar{display:inline;font-size:88%;font-weight:normal}.mw-parser-output .navbar-collapse{float:left;text-align:left}.mw-parser-output .navbar-boxtext{word-spacing:0}.mw-parser-output .navbar ul{display:inline-block;white-space:nowrap;line-height:inherit}.mw-parser-output .navbar-brackets::before{margin-right:-0.125em;content:"[ "}.mw-parser-output .navbar-brackets::after{margin-left:-0.125em;content:" ]"}.mw-parser-output .navbar li{word-spacing:-0.125em}.mw-parser-output .navbar a>span,.mw-parser-output .navbar a>abbr{text-decoration:inherit}.mw-parser-output .navbar-mini abbr{font-variant:small-caps;border-bottom:none;text-decoration:none;cursor:inherit}.mw-parser-output .navbar-ct-full{font-size:114%;margin:0 7em}.mw-parser-output .navbar-ct-mini{font-size:114%;margin:0 4em}html.skin-theme-clientpref-night .mw-parser-output .navbar li a abbr{color:var(--color-base)!important}@media(prefers-color-scheme:dark){html.skin-theme-clientpref-os .mw-parser-output .navbar li a abbr{color:var(--color-base)!important}}@media print{.mw-parser-output .navbar{display:none!important}}

/* end https://en.wikipedia.org/ */

</style><div id="Artificial_intelligence_(AI)426" style="font-size:114%;margin:0 4em"><a href="Artificial_intelligence" title="Artificial intelligence">Artificial intelligence</a> (AI)</div></th></tr><tr><td class="navbox-abovebelow" colspan="2"><div>

<ul><li><a href="History_of_artificial_intelligence" title="History of artificial intelligence">History</a>

<ul><li><a href="Timeline_of_artificial_intelligence" title="Timeline of artificial intelligence">timeline</a></li></ul></li>

<li><a href="List_of_artificial_intelligence_companies" title="List of artificial intelligence companies">Companies</a></li>

<li><a href="List_of_artificial_intelligence_projects" title="List of artificial intelligence projects">Projects</a></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%">Concepts</th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Parameter" title="Parameter">Parameter</a>

<ul><li><a href="Hyperparameter_(machine_learning)" title="Hyperparameter (machine learning)">Hyperparameter</a></li></ul></li>

<li><a href="Loss_functions_for_classification" title="Loss functions for classification">Loss functions</a></li>

<li><a href="Regression_analysis" title="Regression analysis">Regression</a>

<ul><li><a href="Bias%E2%80%93variance_tradeoff" title="Bias–variance tradeoff">Bias–variance tradeoff</a></li>

<li><a href="Double_descent" title="Double descent">Double descent</a></li>

<li><a href="Overfitting" title="Overfitting">Overfitting</a></li></ul></li>

<li><a href="Cluster_analysis" title="Cluster analysis">Clustering</a></li>

<li><a href="Gradient_descent" title="Gradient descent">Gradient descent</a>

<li><a href="Quasi-Newton_method" title="Quasi-Newton method">Quasi-Newton method</a></li>

<li><a href="Conjugate_gradient_method" title="Conjugate gradient method">Conjugate gradient method</a></li></ul></li>

<li><a href="Backpropagation" title="Backpropagation">Backpropagation</a></li>

<li><a href="Attention_(machine_learning)" title="Attention (machine learning)">Attention</a></li>

<li><a href="Convolution" title="Convolution">Convolution</a></li>

<li><a href="Normalization_(machine_learning)" title="Normalization (machine learning)">Normalization</a>

<ul><li><a href="Batch_normalization" title="Batch normalization">Batchnorm</a></li></ul></li>

<li><a href="Activation_function" title="Activation function">Activation</a>

<ul><li><a href="Softmax_function" title="Softmax function">Softmax</a></li>

<li><a href="Sigmoid_function" title="Sigmoid function">Sigmoid</a></li>

<li><a href="Rectifier_(neural_networks)" title="Rectifier (neural networks)">Rectifier</a></li></ul></li>

<li><a href="Gating_mechanism" title="Gating mechanism">Gating</a></li>

<li><a href="Weight_initialization" title="Weight initialization">Weight initialization</a></li>

<li><a href="Training%2C_validation%2C_and_test_data_sets" title="Training, validation, and test data sets">Datasets</a>

<ul><li><a href="Data_augmentation" title="Data augmentation">Augmentation</a></li></ul></li>

<li><a href="Prompt_engineering" title="Prompt engineering">Prompt engineering</a></li>

<li><a href="Reinforcement_learning" title="Reinforcement learning">Reinforcement learning</a>

<ul><li><a href="Q-learning" title="Q-learning">Q-learning</a></li>

<li><a href="State%E2%80%93action%E2%80%93reward%E2%80%93state%E2%80%93action" title="State–action–reward–state–action">SARSA</a></li>

<li><a href="Imitation_learning" title="Imitation learning">Imitation</a></li>

<li><a href="Policy_gradient_method" title="Policy gradient method">Policy gradient</a></li></ul></li>

<li><a href="Diffusion_process" title="Diffusion process">Diffusion</a></li>

<li><a href="Latent_diffusion_model" title="Latent diffusion model">Latent diffusion model</a></li>

<li><a href="Autoregressive_model" title="Autoregressive model">Autoregression</a></li>

<li><a href="Adversarial_machine_learning" title="Adversarial machine learning">Adversary</a></li>

<li><a href="Uncanny_valley" title="Uncanny valley">Uncanny valley</a></li>

<li><a href="Self-supervised_learning" title="Self-supervised learning">Self-supervised learning</a></li>

<li><a href="Reflection_(artificial_intelligence)" class="mw-redirect" title="Reflection (artificial intelligence)">Reflection</a></li>

<li><a href="Recursive_self-improvement" title="Recursive self-improvement">Recursive self-improvement</a></li>

<li><a href="Hallucination_(artificial_intelligence)" title="Hallucination (artificial intelligence)">Hallucination</a></li>

<li><a href="Word_embedding" title="Word embedding">Word embedding</a></li>

<li><a href="Vibe_coding" title="Vibe coding">Vibe coding</a></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%">Applications</th><td class="navbox-list-with-group navbox-list navbox-even" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Machine_learning" title="Machine learning">Machine learning</a>

<ul><li><a href="Prompt_engineering#In-context_learning" title="Prompt engineering">In-context learning</a></li></ul></li>

<li><a href="Neural_network_(machine_learning)" title="Neural network (machine learning)">Artificial neural network</a>

<ul><li><a href="Deep_learning" title="Deep learning">Deep learning</a></li></ul></li>

<li><a href="Language_model" title="Language model">Language model</a>

<ul><li><a href="Large_language_model" title="Large language model">Large language model</a></li>

<li><a href="Reasoning_language_model" title="Reasoning language model">Reasoning language model</a></li>

<li><a href="Model_Context_Protocol" title="Model Context Protocol">Model Context Protocol</a></li>

<li><a href="Intelligent_agent" title="Intelligent agent">Intelligent agent</a></li>

<li><a href="Artificial_human_companion" title="Artificial human companion">Artificial human companion</a></li>

<li><a href="Humanity's_Last_Exam" title="Humanity's Last Exam">Humanity's Last Exam</a></li>

<li><a href="Artificial_general_intelligence" title="Artificial general intelligence">Artificial general intelligence (AGI)</a></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%">Implementations</th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em"></div><table class="nowraplinks navbox-subgroup" style="border-spacing:0"><tbody><tr><th scope="row" class="navbox-group" style="width:1%">Audio–visual</th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="AlexNet" title="AlexNet">AlexNet</a></li>

<li><a href="WaveNet" title="WaveNet">WaveNet</a></li>

<li><a href="Human_image_synthesis" title="Human image synthesis">Human image synthesis</a></li>

<li><a href="Computer_vision" title="Computer vision">Computer vision</a></li>

<li><a href="Deep_learning_speech_synthesis" title="Deep learning speech synthesis">Speech synthesis</a>

<li><a href="ElevenLabs" title="ElevenLabs">ElevenLabs</a></li></ul></li>

<li><a href="Speech_recognition" title="Speech recognition">Speech recognition</a>

<ul><li><a href="Whisper_(speech_recognition_system)" title="Whisper (speech recognition system)">Whisper</a></li></ul></li>

<li><a href="Facial_recognition_system" title="Facial recognition system">Facial recognition</a></li>

<li><a href="AlphaFold" title="AlphaFold">AlphaFold</a></li>

<li><a href="Text-to-image_model" title="Text-to-image model">Text-to-image models</a>

<ul><li><a href="Aurora_(text-to-image_model)" class="mw-redirect" title="Aurora (text-to-image model)">Aurora</a></li>

<li><a href="Adobe_Firefly" title="Adobe Firefly">Firefly</a></li>

<li><a href="Ideogram_(text-to-image_model)" title="Ideogram (text-to-image model)">Ideogram</a></li>

<li><a href="Imagen_(text-to-image_model)" title="Imagen (text-to-image model)">Imagen</a></li>

<li><a href="Midjourney" title="Midjourney">Midjourney</a></li>

<li><a href="Recraft" title="Recraft">Recraft</a></li>

<li><a href="Stable_Diffusion" title="Stable Diffusion">Stable Diffusion</a></li></ul></li>

<li><a href="Text-to-video_model" title="Text-to-video model">Text-to-video models</a>

<ul><li><a href="Dream_Machine_(text-to-video_model)" title="Dream Machine (text-to-video model)">Dream Machine</a></li>

<li><a href="Runway_(company)#Services_and_technologies" title="Runway (company)">Runway Gen</a></li>

<li><a href="MiniMax_(company)#Hailuo_AI" title="MiniMax (company)">Hailuo AI</a></li>

<li><a href="Kling_(text-to-video_model)" class="mw-redirect" title="Kling (text-to-video model)">Kling</a></li>

<li><a href="Music_and_artificial_intelligence" title="Music and artificial intelligence">Music generation</a>

<ul><li><a href="Riffusion" title="Riffusion">Riffusion</a></li>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%">Text</th><td class="navbox-list-with-group navbox-list navbox-even" style="width:100%;padding:0"><div style="padding:0 0.25em">

<li><a href="GloVe" title="GloVe">GloVe</a></li>

<li><a href="Llama_(language_model)" title="Llama (language model)">Llama</a></li>

<li><a href="Chinchilla_(language_model)" title="Chinchilla (language model)">Chinchilla AI</a></li>

<li><a href="ChatGPT" title="ChatGPT">ChatGPT</a></li>

<li><a href="Claude_(language_model)" title="Claude (language model)">Claude</a></li>

<li><a href="Gemini_(language_model)" title="Gemini (language model)">Gemini</a>

<ul><li><a href="Gemini_(chatbot)" title="Gemini (chatbot)">chatbot</a></li></ul></li>

<li><a href="LaMDA" title="LaMDA">LaMDA</a></li>

<li><a href="BLOOM_(language_model)" title="BLOOM (language model)">BLOOM</a></li>

<li><a href="Project_Debater" title="Project Debater">Project Debater</a></li>

<li><a href="IBM_Watson" title="IBM Watson">IBM Watson</a></li>

<li><a href="IBM_Watsonx" title="IBM Watsonx">IBM Watsonx</a></li>

<li><a href="IBM_Granite" title="IBM Granite">Granite</a></li>

<li><a href="Huawei_PanGu" title="Huawei PanGu">PanGu-Σ</a></li>

<li><a href="DeepSeek_(chatbot)" title="DeepSeek (chatbot)">DeepSeek</a></li>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%">Decisional</th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="AlphaGo" title="AlphaGo">AlphaGo</a></li>

<li><a href="AlphaZero" title="AlphaZero">AlphaZero</a></li>

<li><a href="OpenAI_Five" title="OpenAI Five">OpenAI Five</a></li>

<li><a href="Self-driving_car" title="Self-driving car">Self-driving car</a></li>

<li><a href="MuZero" title="MuZero">MuZero</a></li>

<li><a href="Action_selection" title="Action selection">Action selection</a>

<ul><li><a href="AutoGPT" title="AutoGPT">AutoGPT</a></li></ul></li>

<li><a href="Robot_control" title="Robot control">Robot control</a></li></ul>

</div></td></tr></tbody></table><div></div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%">People</th><td class="navbox-list-with-group navbox-list navbox-even" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Alan_Turing" title="Alan Turing">Alan Turing</a></li>

<li><a href="Warren_Sturgis_McCulloch" title="Warren Sturgis McCulloch">Warren Sturgis McCulloch</a></li>

<li><a href="Walter_Pitts" title="Walter Pitts">Walter Pitts</a></li>

<li><a href="John_von_Neumann" title="John von Neumann">John von Neumann</a></li>

<li><a href="Claude_Shannon" title="Claude Shannon">Claude Shannon</a></li>

<li><a href="Shun'ichi_Amari" title="Shun'ichi Amari">Shun'ichi Amari</a></li>

<li><a href="Kunihiko_Fukushima" title="Kunihiko Fukushima">Kunihiko Fukushima</a></li>

<li><a href="Takeo_Kanade" title="Takeo Kanade">Takeo Kanade</a></li>

<li><a href="Marvin_Minsky" title="Marvin Minsky">Marvin Minsky</a></li>

<li><a href="John_McCarthy_(computer_scientist)" title="John McCarthy (computer scientist)">John McCarthy</a></li>

<li><a href="Nathaniel_Rochester_(computer_scientist)" title="Nathaniel Rochester (computer scientist)">Nathaniel Rochester</a></li>

<li><a href="Allen_Newell" title="Allen Newell">Allen Newell</a></li>

<li><a href="Cliff_Shaw" title="Cliff Shaw">Cliff Shaw</a></li>

<li><a href="Herbert_A._Simon" title="Herbert A. Simon">Herbert A. Simon</a></li>

<li><a href="Oliver_Selfridge" title="Oliver Selfridge">Oliver Selfridge</a></li>

<li><a href="Frank_Rosenblatt" title="Frank Rosenblatt">Frank Rosenblatt</a></li>

<li><a href="Bernard_Widrow" title="Bernard Widrow">Bernard Widrow</a></li>

<li><a href="Joseph_Weizenbaum" title="Joseph Weizenbaum">Joseph Weizenbaum</a></li>

<li><a href="Seymour_Papert" title="Seymour Papert">Seymour Papert</a></li>

<li><a href="Seppo_Linnainmaa" title="Seppo Linnainmaa">Seppo Linnainmaa</a></li>

<li><a href="Paul_Werbos" title="Paul Werbos">Paul Werbos</a></li>

<li><a href="Geoffrey_Hinton" title="Geoffrey Hinton">Geoffrey Hinton</a></li>

<li><a href="John_Hopfield" title="John Hopfield">John Hopfield</a></li>

<li><a href="J%C3%BCrgen_Schmidhuber" title="Jürgen Schmidhuber">Jürgen Schmidhuber</a></li>

<li><a href="Yann_LeCun" title="Yann LeCun">Yann LeCun</a></li>

<li><a href="Yoshua_Bengio" title="Yoshua Bengio">Yoshua Bengio</a></li>

<li><a href="Lotfi_A._Zadeh" title="Lotfi A. Zadeh">Lotfi A. Zadeh</a></li>

<li><a href="Stephen_Grossberg" title="Stephen Grossberg">Stephen Grossberg</a></li>

<li><a href="Alex_Graves_(computer_scientist)" title="Alex Graves (computer scientist)">Alex Graves</a></li>

<li><a href="James_Goodnight" title="James Goodnight">James Goodnight</a></li>

<li><a href="Andrew_Ng" title="Andrew Ng">Andrew Ng</a></li>

<li><a href="Ilya_Sutskever" title="Ilya Sutskever">Ilya Sutskever</a></li>

<li><a href="Alex_Krizhevsky" title="Alex Krizhevsky">Alex Krizhevsky</a></li>

<li><a href="Ian_Goodfellow" title="Ian Goodfellow">Ian Goodfellow</a></li>

<li><a href="Demis_Hassabis" title="Demis Hassabis">Demis Hassabis</a></li>

<li><a href="David_Silver_(computer_scientist)" title="David Silver (computer scientist)">David Silver</a></li>

<li><a href="Andrej_Karpathy" title="Andrej Karpathy">Andrej Karpathy</a></li>

<li><a href="Ashish_Vaswani" title="Ashish Vaswani">Ashish Vaswani</a></li>

<li><a href="Noam_Shazeer" title="Noam Shazeer">Noam Shazeer</a></li>

<li><a href="Aidan_Gomez" title="Aidan Gomez">Aidan Gomez</a></li>

<li><a href="Mustafa_Suleyman" title="Mustafa Suleyman">Mustafa Suleyman</a></li>

<li><a href="Fran%C3%A7ois_Chollet" title="François Chollet">François Chollet</a></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%">Architectures</th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Neural_Turing_machine" title="Neural Turing machine">Neural Turing machine</a></li>

<li><a href="Differentiable_neural_computer" title="Differentiable neural computer">Differentiable neural computer</a></li>

<li><a href="Transformer_(deep_learning_architecture)" title="Transformer (deep learning architecture)">Transformer</a>

<ul><li><a href="Vision_transformer" title="Vision transformer">Vision transformer (ViT)</a></li></ul></li>

<li><a href="Recurrent_neural_network" title="Recurrent neural network">Recurrent neural network (RNN)</a></li>

<li><a href="Long_short-term_memory" title="Long short-term memory">Long short-term memory (LSTM)</a></li>

<li><a href="Gated_recurrent_unit" title="Gated recurrent unit">Gated recurrent unit (GRU)</a></li>

<li><a href="Echo_state_network" title="Echo state network">Echo state network</a></li>

<li><a href="Multilayer_perceptron" title="Multilayer perceptron">Multilayer perceptron (MLP)</a></li>

<li><a href="Convolutional_neural_network" title="Convolutional neural network">Convolutional neural network (CNN)</a></li>

<li><a href="Residual_neural_network" title="Residual neural network">Residual neural network (RNN)</a></li>

<li><a href="Highway_network" title="Highway network">Highway network</a></li>

<li><a href="Mamba_(deep_learning_architecture)" title="Mamba (deep learning architecture)">Mamba</a></li>

<li><a href="Autoencoder" title="Autoencoder">Autoencoder</a></li>

<li><a href="Variational_autoencoder" title="Variational autoencoder">Variational autoencoder (VAE)</a></li>

<li><a href="Generative_adversarial_network" title="Generative adversarial network">Generative adversarial network (GAN)</a></li>

<li><a href="Graph_neural_network" title="Graph neural network">Graph neural network (GNN)</a></li></ul>

</div></td></tr><tr><td class="navbox-abovebelow" colspan="2"><div>

<ul><li> Category</li></ul>

</div></td></tr></tbody></table></div></div><div><div class="zim-footer">

This article is issued from <a class="external text" title="Last edited on 2025-07-11" href="https://en.wikipedia.org/wiki/?title=Regularization_(mathematics)&oldid=1299883660">Wikipedia</a>. The text is available under <a class="external text" href="https://creativecommons.org/licenses/by-sa/4.0/deed.en">Creative Commons Attribution-Share Alike 4.0</a> unless otherwise noted. Additional terms may apply for the media files.

</div>

</div></div>

</div>

</main>

</div>

</body></html>